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ABSTRACT 



A method of dis playing hierarchical data. The method 
includes the steps of: retrieving hierarchical network data 
and displaying the hierarchical network data using a per- 
spective three dimensional view. The hierarchical data 
includes a plurality of data elements. Each data element is 
associated with a network device. An apparatus for display- 
ing hierarchical data is provided. The apparatus comprises a 
display device, a memory device storing the hierarchical 
data, and a processor responsive to the memory. The pro- 
cessor includes a graphical display module that displays the 
hierarchical network data using a perspective three dimen- 
sional view. The hierarchical data includes a plurality of data 
elements where each data element is associated with a 
network device; 

15 Claims, 4 Drawing Sheets 
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METHOD AND APPARATUS FOR 
DISPLAYING HIERARCHICAL DATA 
ASSOCIATED WITH COMPONENTS OF A 
SYSTEM 

FIELD OF THE INVENTION 

This invention relates generally to complex systems, and 
more particularly to displaying hierarchical data associated 
with components of such systems. 

BACKGROUND OF THE INVENTION 

Complex systems with many components, such as cellular 
systems, generally require a high degree of maintenance. In 
order to manage the maintenance of each of the components, 
an operations center with a display representing various 
status indicators of the components is usually provided. 
Conventional systems typically generate reports such as 
alarm or traffic reports to provide system operators infor- 
mation regarding the operation of the system Although 
these reports provide valuable information regarding spe- 
cific aspects of the system, they fail to adequately provide an 
overall view of the system. More recently, improved display 
devices and operations centers use graphical user interfaces. 
Such graphical user interfaces typically display two- 
dimensional geographical information, such as a cell site 
view for a cellular communication system. However, due to 
the increased complexity of modern communication 
systems, there is still a need to display a high level view of 
the entire system. Such a high level view would allow 
trouble spots to be more quickly detected and acted upon. 

Accordingly, there is a need for an improved method and 
apparatus for displaying data associated with components of 
a complex system. 

SUMMARY OF THE INVENTION 

In order to address this need, the present invention pro- 
vides a method of displaying hierarchical data. The method 
includes the steps of retrieving hierarchical network data and 
displaying the hierarchical network data using a perspective 
three dimensional view. T he hierarchical data includes, a 
p lurality of_d ala elements where-each data elemen t is 
a ssociated with a network dev ice. 

In accordance with another aspect of the invention, an 
apparatus for displaying hierarchical data is provided. The 
apparatus comprises a display device, a memory device 
storing the hierarchical data, and a processor responsive to 
the memory. The processor includes a graphical display 
module that displays the hierarchical network data using a 
perspective three dimensional view. The hierarchical data 
includes a plurality of data elements where each data ele- 
ment is associated with a network device. 

The invention itself, together with its intended advantages 
will best be understood by reference to the following 
detailed description, taken in conjunction with the accom- 
panying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a block diagram depicting a cellular system that 
may use an embodiment of the present invention. 

FIG. 2 is a block diagram of the system processor and 
display device of FIG. 1. 

FIG. 3 and FIG. 4 are diagrams of an object displayed by 
the display device of FIG. 2. 

FIG. 5 is a diagram of a hierarchical data element defi- 
nition used by the objects in FIGS. 3 and 4. 



2 

FIG. 6 is a diagram of a hierarchical network that may be 
displayed by the object of FIGS. 3 and 4. 

DETAILED DESCRIPTION OF THE 
5 PREFERRED EMBODIMENTS 

Referring to FIG. 1. a communication system 10 includ- 
ing a plurality of components is illustrated. Communication 
system 10 is a cellular communication system including a 
mobile switching center (MSC) 12, a base station controller 

10 (CBSC) 14. an operation maintenance center (OMC) 20. an 
operation support center 32, and a plurality of base trans- 
ceiver stations (BTS) 16. Each of the plurality of base 
transceiver stations 16 is connected to the CBSC 14 via a 
data interface known to those of ordinary skill in the art. 

15 Similarly the MSC 12 is coupled to the CBSC 14 via a 
conventional cellular interface 13. such as the A plus inter- 
face. The CBSC 14 is connected to the OMC 20 via a data 
connection such as a TCP/IP connection. The OMC 20 is 
connected to the operation support system 32 via bridge 30 

20 and via a data network such as a TCP/IP local area network 
The OMC 20 includes a system processor 40. a user station 
22. a multi-user station 24, a display terminal 26. and a 
computer network interconnection system 28. The computer 
network interconnection system 28 is typically a standard 

25 computer data network such as a TCP/IP ethemet network. 
Although both a single user station and a multi user server 
22 and 24, are shown, these servers are optional. 
R eferring to FIG. 2. a more det ailed diagram of the system 

30 processor 40 is provided, 'me sysleWp^tfc es soir*40 includes 
a mfem ory device 100 and a processor 110. The memory 
devicelOO is coupled to the processor 110 7 v iaab"uTll^rThe 
memory device 100 includes a hierarchical network dat a- 
base 120'w hich further includes a status data memory area 

35 122~and a network relationship data area 124."The processor 
110 includes a three dimensional graphical display module 
112. 

In the preferred embodiment, the display module 112 
includes hardware and software necessary to present the 

4Q hierarchical data to the graphical display. E xampl es of 
suitable s oftware for thedisplay modules 112 include three 
dimension rending software avada^Ie^from^Alias/Wavefxont, 
Virtual reality modeling language (VRML), and HTML/ 
JAVA®. JAVA® is a registered trademark of Sun 

45 Microsystems. Inc. Each of these software packages arc 
known to those of ordinary skill in the art Suitable hardware 
includes a high speed graphics monitor capable of quick 
three dimensional rendering which are known to those skill 
in the art. 

50 The system processor 40 is connected to a display device 
50 in a conventional manner. The display device 50 may be 
used as the display 26 within (he OMC 20 or may be 
incorporated or attached to the operation support system 32. 
In any event, the display device 50 is used by operations 

55 personnel such as technicians, to control and maintain the 
communication system 10. 

The hierarchical network data base 120. including both 
the status data area 122 and the network relationship data 
area 124. contains specific information regarding each of the 

60 components within the communication system 10. For 
example, many components within the base stations 16 such 
as transceivers, may each have an entry within the hierar- 
chical data base 120. In addition, since each component in 
the communication system 10 is typically connected to one 

65 or more other components, the relationship between com- 
ponents is stored in the network relationship data area 124. 
For example, since BTS 16 is coupled to CBSC 14 a network 
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relationship between the BTS 16 and CBSC 14 would be tional resolution display, such as a CRT. it is contemplated 

stored in the network relationship data area 124. In this that up to 750 transceivers may be illustrated. However, it is 

manner data regarding each component in the system 10 as preferred to use a high resolution display, such as a 1280 by 

well as the relationship between components in a hierarchi- 1024 pixel display, so that several thousand transceivers may 

cal model may be stored within the hierarchical network 5 be viewed. In addition to the display of status information, 

data base 120. the display system allows an operator to select one or more 

Referring to FIG. 6. an example of a hierar chical netwo rk of the displayed network target devices, such as a particular 

that may bei mplemente3 with a su itable da tabasTtfaat sto res transceiver, by clicking on the surface area associated with 

mrneTworEMany such databases are known to those of that transceiver, such as by using a computer mouse. After 

ordinary skill in the art For illustrative purposes, an 10 a specific target network device has been selected, many 

example of a database record for one element of information data base type operations may be performed, such as adding 

for one device in such a database is shown in FIG. 5. The a new device, editing perameters on the device* activating a 

particular database record of FIG. 5 describes the telephony device, or deleting a particular device. The necessary per- 

status for a particular device in a cellular system. ameters for the selected device may be provided either 

Referring to FIG. 3. an example of an object shown on 15 trough keyboard data entry or through selection via a 

display device 50 via a three dimensional perspective view mouse. 

is illustrated. The displa y device 50 shows a d isplayed object To provide the use r with further compre hension ofjhe 

60 which preferably h as a three d imensional shape. A system, a Umnhty uf hoi unx accesses from-a-pafticular 

plurality of network devices 62. 64. 66 are each represented" nelworfc-device^o-rel^ of the particular^ 

b vluTreient^ 20 ^^^"gtl ^ a detail ed diagram or photographof the 

60. In addition, relationships-between networ k devic es ma y devi ce, may be provided. T h e detai led jiag ram or ph oto- 

be showjuon-th£_ displayed ^ob ject 60. such -a&-iiy_having graph may be supplemented" by explodedHiagrams of th e 

hig^eiJewljietwctfkd^ices 6 1 displayed on an in ner region devic e and how that device interconnects as well as a textual 

of^^^^j^^j^ ^aj^TKMouSi^^^spU^^ ob ject 60 expl anation of the purpose and capacity of e^h~such"devicer 

is shown_to be stationar y, it is contemplated tfiaTtlie~dis ^ 2 5 A£^S^S5^yifcdjt^e^playris-mat a database 

plaved obieel may be rotated lnany jlirection by command q uery may be^ma de^of the s y$tem ~to~show*all networ k 

of an operator^u^h-mat me"^era1or may view the.entire devices tha^poss ess cert ain qualities. For example, all 

surface~of mlTdispiay^ a greater dev ices that satisfy a particular query, such as al l active 

numbed of^etworlTaevices and their associated relation- devices on a particular base station 16 or connected'tiT a 

ships with other network devices may be quickly seen at a 30 particuja rGBSC 14. may ap pear in a designated co lor. This 

high level. In addition to showing each network device, fe ature may be particularly useful in detecting defects in a 

information associated with each network device may be net work device such a s a t ransceiver, bef ore it fails com- 

displayed on the surface of the displayed object 60. For pletel y. Another anticipated feature of the disp lay d^iceSO 

example, a status indicator of each network device may be and uiejystem processor 40 ^ that r equested additions or 

shown via a different color on the surface region for each 35 deletions maae bylhe op erator may bVsyntHiesizcd'into a 

respective network device. For example, an out of service parts4ts tsocn as an order form for new eq ui pment B y using 

network device may be shown in red. an in service device the displaydevice 50 and operator conu nands to form a price 

may he shown in blue, and a device that has not been ^list rapid quotes may be_made„a vailable,to tr^customer. 

provisioned may be shown in green. Another method of Although ma nyjfts^lay^e^ces.ma y be used, iUsjgef erred 

displaying information regarding each network device 40 tha t the device should be sui ted tojhreejymensjo^^ 

would be to use a flashing indicator to display alarm toring. An e xample of a suitable display device js _ajugh 

conditions. speed graptucs worKstation. such as those made by Silicon 

Referring to FIG. 4, a particular exemplary displayed GrapMcOnc^ " " ~ 

object for a cellular communication system is illustrated. Although FIG. 4 refers to the network devices of com- 

The displayed object 60 has a spherical shape, only a cross 45 munication system 10 which is a cellular communication 

sectional slice of the sphere is shown in FIG. 4, and includes system, the present invention is applicable to many noncel- 

a p lurality of network devices, which in this particular case lular applications. For example, other communication net- 

arelransc eivers, on the outsi de su rface o f t he object 60. .On work such as telephone networks, data networks or satellite 

th ft innCT su rface, which is conical in this particular example, networks may advantageously use the method and apparatus 

a p lurali ty of highe r level hierarc hical relate d networ k 50 discussed above. In addition other systems such as utilities. 

deviceOssraal^"with the t ranscavwOT me^urface are transportation networks and assembly line or manufacturing 

shown. For exam ple, in the c eUular.cornmuni cation s ystem, . environments could make use of displays such as those 

thejOMC20 is a high level network device and is posi tione d described herein. 

in the core of tfie~comc al area .62. X^er.Detw^^devices. in The illustrative method and apparatus described herein 

theTsysteln^O-^ above OMC 20 Lm.a concentric 55 have many advantages. For example, the apparatus provides 

fashion. For example; otKer rier work devices include mob il- the ability for a technical operator to quickly observe and 

ity managers labeled MM. trans coders l abeled XC. which monitor the status of any one of many, components, such as 

are typically foundlmthin.CBSC 14. a front end processor transceivers and associated elements in a complex system. 

(FEP), a multipl e s pan line i nterface processor (MSDP), a Consequently, the operator may more quickly detect and 

BTSJfcrfc^sjuc^s^tiie.link.te 60 react to trouble spots in the system. In addition, system 

an d a master group line i nterface (MGLI). " operators may more quickly gain familiarity with the system 

All of the transceivers which would be found in a par- due to the three dimensional user friendly presentation of 

ticular BTS 16 are preferably arranged in a vertical segment system components. 

that extend along a longitudinal line over me surface of the Further advantages and modifications of the above 

displayed object 60. In this manner, an operator may imme- 65 described apparatus and method will readily occur to those 

diately know which transceivers are associated with a par- skilled in the art The invention, in its broader aspects, is 

ticular BTS 16. Where the displayed device 50 is a conven- therefore not limited to the specific details, representative 
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apparatus, and illustrative examples shown and described 
above. Various modifications and variations can be made to 
the above specification without departing from the scope or 
spirit of the present invention, and it is intended that the 
present invention cover all such modifications and variations 
provided they come within the scope of the following claims 
and their equivalents. 
What is claimed is: 

1. A method of displaying hierarchical data comprising 
the steps of: 

retrieving hierarchical network data representative of a 
status of a network, said data including a plurality of 
data elements, each data element associated with a 
network device; and 

displaying the hierarchical network data onto a surface of 
a three dimensional structure, displayed on a two- 
dimensional display for the purpose of controlling the 
network based on the display. 

2. The method of claim 1. wherein said structure com- 
prises a conical object. 

3. The method of claim 1. wherein said structure is 
selected from the group consisting essentially of spherical 
objects and concentric circular objects. 

4. The method of claim 1, wherein said structure changes 
position over a period of time. 

5. The method of claim 4. wherein said structure changes 
position in a tumbling pattern over said period of time. . 

6. The method of claim 1. wherein said surface has a 
plurality of display regions, at least some of said display 
regions representing at least some of the data elements. 

7. The method of claim 6. wherein at least one of said 
plurality of display regions is uniquely selectable. 

8. The method of claim 7, wherein further information is 
associated with the network device linked to the data ele- 
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ment represented by said at least one display region when 
said at least one display region is selected. 

9. An apparatus for displaying hierarchical, data compris- 
ing: 

5 a display device; 

a memory device storing the hierarchical data represen- 
tative of a status of a network, said hierarchical data 
including a plurality of data elements, each data ele- 
ment associated with a network device; and 

io a processor responsive to said memory and including a 
graphical display module that displays the hierarchical 
network data onto a surface of a three dimensional 
structure, displayed on a two-dimensional display 
device, for the purpose of controlling the network 

15 based on the display. 

10. The apparatus of claim 9. wherein said network device 
comprises a communication system network device. 

11. The apparatus of claim 9. wherein said structure 
comprises one of a conical object and a spherical object. 

12. The apparatus of claim 9. wherein said structure 
20 position on the display device over a period of time. 

13. The apparatus of claim 9. wherein said surface has a 
plurality of display regions, at least some of said display 
regions representing at least some of the data elements, 

14. The apparatus of claim 13. further comprising a 
25 pointing device in communication with said display device, 

and wherein at least one of said plurality of display regions 
is uniquely selectable by said pointing device. 

15. The apparatus of claims 9, wherein said memory 
device comprises a first memory area storing status data 
associated with each network device and a second memory 
area storing network device relationship data defining a 
relationship between the data elements. 

* * * * * 
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ABSTRACT 



A limitation of using two-dimensional images, such as 
videos or photographs , to represent portions of a three - 
di mensionaTwona o ccurs when the user moves within trie 
worl d and views the world from a location di fferent than 
fronTthe original context of the two-dunensionaTimage, i!e ., 
f rom a location different than the_i ma g e's ideal viewin g 
point (I VP). View changes result in the image not aligning 
well with the surroundiS g oHiects of llie^ hree^im^nsional 
world. This l imitation is overcome by d istorting the~twb~ 
dimensio nal"*image so as to adjust th e image's vanishing 
p oint(s)Tn~accorda nce with the movem^t^fHh^useTusin g 
a pyramidic panel structure .. In^this manner, as the user 
moves away from the ideal viewing point, the distortions act 
to limit the discontinuities between the two-dimensional 
image and its surroundings. To minimize the depth profile of 
the pyramidic panel structure, the structure may be seg- 
mented into sections and each section translated towards, or 
away from, the user's viewpoint. Also, a hierarchical image 
resolution may be used, with portions of the image near the 
center or vanishing point having a higher resolution than the 
portions of the image near its perimeter. 

15 Claims, 14 Drawing Sheets 



QnterX 130 



DETERMINE VECTOR C CORRESPONDING TO DIRECTION 
OF THE ROAD, PROJECTED FROM WP TO PANEL OR SCREEN 



-135 



SEGMENT PANEL INTO FOUR TRIANGULAR REGIONS WITH INTERSECTION 
POINT OF THE FOUR REGIONS AT VANISHING POINT, P 



DETERMINE CURRENT VIEWPOINT, V, OF THE USER 
AND PROJECT VECTOR, T FROM IVP TO V . 



CALCULATE NEW VANISHING POINT, P'=P+T AND DISTORT 
THE FOUR TRIANGULAR REGIONS SUCH THAT THEY INTERSECT 
AT POINT, P* 



TEXTURE MAP IMAGES DISPLAYED IN REGIONS 145j-1454 TO 
PYRAMIDIC PANELS 145 j -145*4 



-150 



-155 



-160 



10/16/2003, EAST version: 1.04.0000 



U.S. Patent Jan. 7, 2003 Sheet 1 of 14 US 6,504,535 Bl 




10/16/2003, EAST version: 1.04.0000 



U.S. Patent Jan. 7, 2003 Sheet 2 of 14 US 6,504,535 Bl 



FIG. 3 



CE 



130 



DETERMINE VECTOR C CORRESPONDING TO DIRECTION 
OF THE ROAD, PROJECTED FROM IVP TO PANEL OR SCREEN 



SEGMENT PANEL INTO FOUR TRIANGULAR REGIONS WITH INTERSECTION 
POINT OF THE FOUR REGIONS AT VANISHING POINT, P 



DETERMINE CURRENT VIEWPOINT, V,. OF THE USER 
AND PROJECT VECTOR, T FROM IVP TO V 



CALCULATE NEW VANISHING POINT, P'=P+T AND DISTORT 
THE FOUR TRIANGULAR REGIONS SUCH THAT THEY INTERSECT 
AT POINT, P' 



TEXTURE MAP IMAGES DISPLAYED IN REGIONS U5i~1454 TO 
PYRAMIDIC PANELS 145] -145 4 



135 



140 



150 



155 



160 



CEO 165 



10/16/2003, EAST version: 1.04.0000 



U.S. Patent 



Jan. 7, 2003 



Sheet 3 of 14 



US 6,504,535 Bl 



FIG. 4 

y 



110 




10/16/2003, EAST Version: 1.04.0000 



U.S. Patent 



Jan. 7, 2003 Sheet 4 of 14 



US 6,504,535 Bl 




FIG. SB 

lis m 




125 



10/16/2003, EAST Version: 1.04.0000 



U.S. Patent 



Jan. 7, 2003 



Sheet 5 of 14 



US 6,504,535 Bl 



FIG. 7A 

105 




125 



FIG. 7B 



115 105 




125 



10/16/2003, EAST Version: 1.04.0000 



U.S. Patent 



Jan. 7, 2003 Sheet 6 of 14 



US 6,504,535 Bl 




10/16/2003, EAST version: 1.04.0000 



U.S. Patent 



Jan. 7, 2003 Sheet 7 of 14 



US 6,504,535 Bl 



FIG. 9 



CENTER 



170 



DETERMINE DIRECTION AND LENGTH VECTORS Cj, C 2 
OF THE TWO PORTIONS OF THE ROAD 



DETERMINE THE VANISHING POINTS Pj, P 2 FOR EACH 
PORTION OF THE ROAD 



CONSTRUCT FIRST PYRAMIDIC PANELS INTERSECTING 
AT VANISHING POINT, ^ 



175 



180 



185 



CALCULATE COUPLING RATION, a 



V-195 



I 



CONSTRUCT SECOND SET OF PYRAMIDIC PANELS MOVED 
FROM Pi TO P2, WITH BASE CORNERS Q1-Q4 



I 



DETERMINE CURRENT VIEWPOINT OF USER AND 
PROJECT VECTOR T FROM IDEAL VIEWING POINT, 
IVP, TO CURRE NT VIEWPOINT LOCATION, V 

I 



CALCULATE NEW VANISHING, P 2 ACCORDING TO 
P 2 =P 2 +T, WITH SECOND SET OF PANELS 
DISTORTED TO INTERSECT AT P' 2 

I 



TRANSFORM INTERNAL POINTS Q.1-Q4 
ACCORDING TO Qj =Qj+aT 



TEXTURE MAP IMAGES ONTO ARTICULATED 
PYRAMIDIC PANELS 

I 



nr 



230 



205 



210 



215 



220 



225 



10/16/2003, EAST version: 1.04.0000 



U.S. Patent Jan. 7, 2003 Sheet 8 of 14 US 6,504,535 Bl 

FIG. 10 



105 




^3 200 3 



FIG. 11 



105 




10/16/2003, EAST version: 1.04.0000 



U.S. Patent Jan. 7, 2003 Sheet 9 of 14 



US 6,504,535 Bl 



FIG. 12 




T 2 4- T 



10/16/2003, EAST Version: 1.04.0000 



U.S. Patent Jan. 7, 2003 Sheet 10 of 14 US 6,504,535 



FIG. 13 




10/16/2003, EAST version: 1.04.0000 



U.S. Patent Jan. 7, 2003 Sheet 11 of 14 



US 6,504,535 Bl 



FIG. 14 




T 2 4- T H 



10/16/2003, EAST version: 1.04.0000 



U.S. Patent Jan. 7, 2003 Sheet 12 of 14 



US 6,504,535 Bl 




10/16/2003, EAST Version: 1.04.0000 



U.S. Patent 



Jan. 7, 2003 Sheet 13 of 14 

FIG. 16 



US 6,504,535 Bl 




10/16/2003, east version: 1.04.0000 



U.S. Patent Jan. 7, 2003 Sheet 14 of 14 US 6,504,535 Bl 



FIG. 18 



110, 115 



215 4 
"2157" 



1114 x m4 
iri3 x 1113 



2152 m2 


x m2 




21 5 1 mi x m\ 






• 






P 







10/16/2003, EAST version: 1.04.0000 



US 6,5' 

1 

DISPLAY TECHNIQUES FOR THREE- 
DIMENSIONAL VIRTUAL REALITY 

CROSS-REFERENCE TO RELATED 
APPLICATION 

This application is related to co-pending U.S. patent 
application Ser. No. 09/190,743 filed concurrently herewith 
(Case Edmark-7). Also, this application is a continuation- 
in-part of U.S. patent application Ser. No. 09/160,758 filed 
Sep. 25, 1998 (Case Edmark-5), which is a continuation-in- 
part of U.S. patent application Ser. No. 09/107,059 filed Jun. 
30, 1998 (Case Edmark-2). The above -identified co-pending 
applications, which are commonly assigned, are incorpo- 
rated herein by reference. 

TECHNICAL FIELD 

This invention relates to the integration of three- 
dimensional computer graphics and a two-dimensional 
image to provide a realistic three-dimensional virtual reality 
experience. 

BACKGROUND OF THE INVENTION 

The display of a three-dimensional virtual reality world to 
a user requires considerable computation power, and it is 
typically costly to develop the necessary highly detailed 
models required for doing so. In order to simplify the 
problem, two-dimensional images, such as videos or 
photographs, may be used to represent or simulate portions 
of the three-dimensional world. A great reduction in com- 
putation power and cost can be achieved by such an arrange- 
ment. 

SUMMARY OF THE INVENTION 

A limitation of such a world occurs when a user moves 
within 'the world ana views trIe~"woTld~from a location 
different than the original context of a two-dimensional 
image which has been carefully calibrated to "fit into" the 
world. View changeSjSuch as from a location different than 
the ima gg*TlQ^ljviem not 
aligning or fittin g well witlTthe suiroun dingo5iects~of the 
three^dimensional_wj3rld._ Lhave recognizeg ~triat7"in"accor- 
dance^witht he principles of the in vention, viewpoint 
changesma v De deal t with b y distortin g, the.twordimensional 
image so^^io^^x^isWhe^mage-s-vanishing-point^) in 
accordance wi th the movement o f the user using a novel 
"pyramidic panel structure. " In truT"mannerr as- the- user 
moves away-from'thBidealViewing point, the distortions act 
to limit 4he-discoTilinuities bet ween the two-dime nsional 
image andahe-su TruundwgsroTthe world. 

In another aspect of the presen t invention, the pyramidic 
panel struct ure"may"be se gmented'i nto sections ,~ ~each trans- 
lated toward s or away from the user J s_vie.w pofnt,and~then 
scaled, so as to.jninimize*the^ptJi^rofile of the pyramidic 
panel structure. I n vet still anomeras pe cTof "the prese nt 
i nvention, a hierarchical image resolution may, beused, with 
po rtions of the image near t he center of the image having a 
h ig Her resolution than the portions of'trie^imag e^n^aP it's 
perirnglsr, ~" 

BRIEF DESCRIPTION OF THE DRAWINGS 
In the drawings: 

FIG. 1 shows an example of that which a user sees when 
a user views the world from the ideal viewing point for a 
two-dimensional image representing a portion of the world; 



34,535 Bl 

2 

FIG. 2 shows an example of that which a user sees when 
a user moves within the world of FIG. 1 and views the 
two-dimensional image from a location different than the 
image's ideal viewing point, without the use of the present 
5 invention; 

FIG. 3 shows an exemplary process, in accordance with 
the principles of the invention, for distorting the two- 
dimensional image using a pyramidic panel structure so as 
to adjust the image's vanishing point in accordance with the 
10 movement of the user; 

FIGS. 4 and 5 depict the pyramidic panel structure of the 
present invention for distorting the two-dimensional image 
so as to adjust the image's vanishing point, in accordance 
with the movement of the user: 

15 • 

FIGS. 6A-B depict examples of that which a user sees 
when a user views the world from a location left of the 
image's ideal viewing point, without and with the use of the 
present invention, respectively; 
20 FIGS. 7A-B depict examples of that which a user sees 
when a user views the world from a location above the 
image's ideal viewing point, without and with the use of the 
present invention, respectively; 

FIGS. 8A-B depict examples of that which a user sees 
25 when a user views the world from a location toward the front 
and the right of the image's ideal viewing point, without and 
with the use of the present invention, respectively; 

FIG. 9 shows an exemplary process, in accordance with 
the principles of the invention, for distorting a two- 
30 dimensional image using an articulated pyramidic panel 
structure so as to adjust multiple vanishing points in the 
image, in accordance with the movement of the user; 

FIG. 10 depicts an example of the articulated pyramidic 
panel structure of the present invention; 

FIG. 11 depicts an example of that which a user sees when 

a user views the world from a location away from the ideal 

viewing point of the two-dimensional image, with the use of 

the articulated pyramidic panel structure of the present 

An invention; 
40 * 

FIGS. 12 and 13 depict side and front views, respectively, 
of the pyramidic panel structure of FIG. 5 with each panel 
segmented into a plurality of sections; 

FIG. 14 depicts the pyramidic panel structure of FIG. 5, 
45 with each panel segmented into a plurality . of sections 
having its centers located on the surface of a predetermined 
plane; 

FIG. 15 depicts the pyramidic panel structure of FIG. 5, 
with each panel segmented into a plurality of sections and 
50 each section translated a different distance toward the user's 
view point, V; 

FIG. 16 depicts the screen of the pyramidic panel struc- 
ture of the present invention segmented into four triangular 
sections with the corresponding portions of a two- 
55 dimensional image displayed in each panel; 

FIG. 17 depicts the screen of FIG. 16 from a location 
closer to the center of the image, with the corresponding 
portions of the image textured-mapped onto the pyramidic 
60 panels in accordance with the present invention; and 

FIG. 18 depicts the screen of FIG. 16 with the two- 
dimensional image segmented into a plurality of sections in 
accordance with another aspect of the present invention. 

DETAILED DESCRIPTION 

65 

To better understand the invention, FIGS. 1-2 show 
examples of that which user sees when the user moves 
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within a three-dimensional virtual reality world (x,y,z) and environment of CG Part 120. This unique location is called 

views a two-dimensional image (x,y) representing a portion the image's ideal viewing point (IVP). In FIG. 1, two- 

of the world from a location at the image's ideal viewing dimensional image 115 is seen from its ideal viewing point, 

point (IVP), and then from a different location, i.e., a and from this view, image 115 aligns well with the surround- 

location different than the original context of the image. It 5 ing objects of CG Part 120. 

should be understood that the two-dimensional image has Users, howev er, rarely view image 115 only from i ts idea 

been carefully calibrated to "fit into" the surroundings of the viewing"point. As the user moves within world life, such as 

world. For simplification of terminology purposes, we shall left or right of road 125, as they round curves, or move closer 

use the term two-dimensional image to denote either a video to or farther from the image, they see image 115 from 

clip or a photograph. In accordance with the principles of the JQ positions other than its ideal viewing point. Absent the use 

invention, as the user moves away from the ideal viewing of the present invention, such viewpoint changes would 

point, discontinuities between the two-dimensional image cause objects or features within image 115 to align improp- 

and its surroundings are minimized by distorting the image erly with the surrounding environment, as further illustrated 

according to the movement of the user. in FIG. 2. 

FIG. 1 sh ows an exemp la ry three-dimensional realit y 35 In accordance with the principles of the invention, 

world 105, which is a bicycle path in a park , e.g., Central however, s creen or panel 110 uses a display structure called 

Park in"New York City. In representin g w orlcTlO S, trie a "pyramidic panel structure" tor displaying - tWO- 

preseht invention exploits a characteristic" common for dimensional image 115~within~th e~surroTIfiding three- 

images^c onsisTing of "views luoKing~clown the center^ of dimensional s pace of the CG Part 105 so as ~to~deal*'with 

roads, streets or paths, which is that they may be treated as 20 viewpoint- changes. The transformations associated with th e 

persp^^iv^7^o1rridor-like"ima'ge s7 witrTfeatures closer tolh e py rarrrjdic_p.anel stru ctti re— dynamical! y— distort— two- 

center~of Irie^inTa' ^b^in^farther away from the viewe r di mensional im age 115 according to viewer's position so as 

along the z-axisrAccordin glv,Jhe_bicvcle_path,or.road__and to adjust the i mage's vanishing point~with~the viewer! !* 

its imme diate vicinity are treated as a kind of three- movementrA Tlhe viewer moves from the ima gelsjdeal. 

dimensional, corridor-like image who se flooTis"for me^By 25 viewmg"po1n[ rthes e_distortiQns,act_to, l imit discontinu ities 

the roadbedrwhose'c e^n^irformed'b y_the_sky,md.whose betweerT image 115 and the surroundin gS-QLCG-Part-120. 

sidewall s are'formed'by the roadside objects. In this manner, "FIG73 sho ws an exemplar y process in ac cordance" with 

the principles of a simple point perspective can be used for the principles of the invention for dis torting two- 

distorting ttielandscape ima gejn accordance withlhe m ove- dimensional-image-ll-S'So^as to adjust its vanishing point in 

ment of the vie wer, as discussed h erein below. 3Q accordance with the viewer's po sition. The process is 

W orld 105 js divided into two portions, screen or panel entered-at-ste p^~130~ whenever J Os_determined--thaMhe 

110 on which is^Kown~or~displayed~a~two^imensio nal viewer's position has changed. 

image 115, such as a still p hotograp hTpictur e, or a cur rent Using th e virtual world'sToad model of the CG Part 105, 
frame of a video cli pj and the remainder of the world 120, a vectoTT C, correspondin g to the direction of road 12 5 is 
which is rep resented usin g computer graphics Jechniques, 35 projected at step 135 from the image's ideal viewmg_point, 
and is thus referred jo herein as computer g ra phics ( CG Part) I W/topanel'or'screerfllO'oh" which'isciisplayecilmage.115. 
120. Within CG ^arjJL20jhere . are var ious s ynthetic, J hree - Not e thafth e paneHs : two-dim ej3sional7but : lipresents three- 
dime nsionatia^a scapes or objects modeledln , fbr.example, dimensional space with objects nearer the center of the 
the Virtual~ReaEty ModeUn g_Unguage_(yRML).--Two- imageT5eing*fafther away fr*o^tlieptane-of theviewer.-The 
dimensionarimage 115 ^ijmilale^landscape.orJerrain-por- 40 panel-stmcture-is showninJTO^ 

tions of the worl dlOS, here a virtual road or course 1 25 for with- screen- or panerilO lslhe ima gejs_yanishiiuz-point w P. 

walking, running oj^ pedalin g a bi cycle. Note, however,~tharth"e~vanishing p oint may be set visua lly 

Note that althoughthre e^ime nsiona l worl d 105cannot be by toeTiser r tf-desired£0xbyJft 

actually rendered in ajwo-dimensional plane (x,y) J jt_can be processing'techmQu es known in_t he.art.-Nexfc-in-step-140.. 

projectedlb" ahdTusplayed on a two-dimensional plane so as 45 screenbr panerilO'is se gmented^into.four-triangular-regions 

to appeaFto±aveahree_dimensjoj^(x,y,^^ 145^ , one for each of the re gions^bordering-CG-Eart-^O, 

techni ques of the present.invejit ion are preferabl y employed with the in tersection point of the fou r_regions-loGated-aMhe 

with comp uters and s oftware, whichjnre sufficiently sophis- vanishing p oint, P. 

ticated to disp lay images on a two-dimensio nai"plane as Thereafter in step" 150, the current viewpoint of the user, 

having thre e dimensions, fto te also that to make"the~ world 50 V, is determined, and a vector T projected from the ideal 

look realistic, comp uter grapKicTdispla y techniquesuse-the viewing point, IVP, to the viewer's current location, V. In 

z component of objects to scale accordingly the x and y accordance with the principles of the invention, as the 

components as a fimction„of.iMistance_(zraxis).to the user's viewer moves, a new vanishing point P' is calculated as 

viewpoint. P'-P+T. The four triangular regions 145^ are distorted in 

TV/o-dimensiona l image 115 is carefully placed, cropped 55 the three-dimensional space of the virtual world at step 155 

and jjzelt_^o_aeM eve^ to represent the mapping of objects nearer the center of the ■ 

ronment of the CG Part 120. Note that the image is clipped image being displaced farther away from the viewpoint of 

so thaTthe left and"Tighre^ges"of"the"?o^a"irrCG"Parri20 the user. The four triangular regions intersect at the new 

pass through the left and right bottom comers of the road, vanishing point P 1 and form so-called "pyramidic panels" 

respectively, in image 115. This clipping ensures that the 60 145^.4. This is illustrated in FIG. 5. At step 160, the 

roadbed maps to the floor of the hypothetical corridor. In so corresponding images displayed in regions 145^ are then 

doing, portions at the boundary between two-dimensional "texture-mapped" onto pyramidic panels 145' 1-4 respec- 

image 115 and CG part 120 are co-planar, i.e., at the same tively. In this manner, as the viewer moves away from the 

distance away along the z-axis from the user's viewpoint. In image's ideal viewing point, IVP, distortions in the image 

"fitting" two-dimensional image 115 to CG part 120, 65 resulting from moving the image's vanishing point from P to 

however, there exits only one viewpoint from which that P' act to limit the discontinuities between image 115 and the 

image's content properly corresponds to the surrounding surroundings within CG Part 105. 
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In the exem plary illustration of FIG . 5, distorting imag e 
1 15 so as to move tne vamstung p oint tr om F to F' results i n 
pyramidic panel structure forming a four-sided pyramid. 
NfMc^thaj its hase is fixed and corresponds to original scree n 
or panel 110, with i ts peak located at P. which moves i n 5 
con cert wan trie viewer' s current location r V. As the user's 
viewpoint m oves closer to and farther from the imag e, the 
image' s vanishing point accordingly moves farther from and 
-closer to the user's viewpoint, respectively. 

FIGS. 6 through 8 compare the dis play of two- 10 
dimen sional irr| a ge"JiA on screen or panel "Su^witfT the 
displav j)f the same image using thT'ffiramiaic" panelsof 
the pre sent invention. More speciflcally^ FIGS. 6A, T/TarTd 
8A^epict \aewing jwikdimen sional ima ge~115"at a location 
from the left 7 above .-and-.in^front^and_to,the^right of the 15 
image~*slHeal viewinj ^omt ,jyP,.respectiyely, without the 
use of thcpresent invention. In these latter figures, note that 
there are discontinuities between the edges of the road and 
the three-dimensional space of CG Part 105. FIGS. 6B, 7B 
and 8C depict the same two-dimensional image distorted 20 
and texture-mapped onto pyramidic panels 145\_ 4 , in accor- 
dance with the principles of the invention. Note that in these 
latter figures, the discontinuities in the road edge have been 
substantially eliminated. 

In another embodiment of the present invention, a modi- 25 
fied pyramidic panel structure may be used to deal with 
two-dimensional images containing curved roads, streets, 
paths and other corridor-like images containing multiple 
rather than a single vanishing point. In this latter case, screen 
or panel 110 is segmented using multiple vanishing points to 30 
form a so called "articulated pyramidic panel structure." The 
tran sformations associated' with the articulated p yramidic 
panel structure dynami c ally distort different por ti ons of 
tvvo^imensional imag eJl5^ccflr3in g,to viewer positions so 

as tO adjust the different va nishing pmnts.nf-frhft.image*with 35 

the "view er's movemeaL.IJk^w.ise.^.theAuawer-mQves.from 
the image's ideal viewmgjjoint , these distortions act to lim it 
the mscontimHtie^^ 
the surrowKfihgs of CG Part 120. 

■ 40 

FIG. 9 shows an e xemplary proces s in accordance with 
the principles nt the— inden tion for distorti ng two- 
dimensionai image 115 using an artic ulate4 pyrami(iic.pa"hel 
structure; Again, the process is en tered at st ep.l7.0jwhenever 
it b-cte tennmedHha t-the-viewer's position has changed. In, 45 
generate curve ro a3~l*25~is"treate*d as two s traight corridors, 
pl ace~d'el^-tOj£ ndZe^ screen .or.panelJL L0. 
Each corridor re presents a different portion of road 125 in 
thelhree- dimensional s pace_b f2w^fld~105r^th~feature s 
nearer" t he cent erjjf.the.image-being-farthe r^away from th e 5Q 
user'sTiewpoint.- — 

Dsing the virtual world's road model of the CG Part 105, 
corresponding directional vectors Cj and C 2 of the corridors 
are determined at step 175. Note that portion of the road 
nearer to the user's viewpoint is represented by C ly and the 55 
portion farther away is represented by C 2 . Next, in step 180, 
using the vectors Q and C 2 , the corresponding vanishing 
points P a and P 2 are determined, respectively, for each 
corridor by projecting those vectors from the image's ideal 
viewing point, I VP. Alternatively, vanishing points P A and P 2 60 
may be determined visually by the user, or by some other 
suitable means known in the art. In step 185, using the first 
corridor's vanishing point, P lf a first set of pyramidic panels 
190 1-4 are constructed to intersect at vanishing point, P lt as 
shown in FIG. 10. 65 

Now at step 195, a coupling ratio a is calculated accord- 
ing to the following equation: a-l/(l+d), where 1 is the length 
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of the first corridor, and d is the distance between the 
image's ideal view point (I VP) and the base of pyramidic 
panels 190^4. Each line segment connecting a corner of the 
base to vanishing point P a is then divided into two segments 
by a point placed according to the coupling ratio, a. More 
specifically, the length 1' of each line segment from the 
corner of the base of panels 190 a . 4 to this point is given by 
T-cd", where 1" is the total length of the segment between the 
corner, of the panel and the vanishing point, P x . These four 
points labeled Ql through Q4 are connected to form the base 
of a second set of smaller pyramidic panels 200j_ 4 embedded 
within the larger panels (step 205), as further illustrated in 
FIG. 10. The intersection point of pyramidic panels 200 1 . 4 
is then moved from P : to vanishing point, P 2 . 

For the articulated pyramidic panel structure, the current 
viewpoint of the user, V, is determined, and a vector T 
projected from the ideal viewing point, IVP, to the viewer's 
current location, V (step 210). As the viewer moves, a new 
vanishing point P 2 is calculated as P 2 '=P 2 +T at step 215, and 
panels 200 1 . 4 are then distorted so as to intersect at P 2 \ As 
the viewer move, the four internal points Ql through Q4 are 
mapped with the viewer's movement to Ql' through Q4\ 
respectively, in accordance with the following relationship: 
Q/'-Qi+aT, at step 220. Note that doing so, accordingly 
distorts the first set of pyramidic panels 190 2 _ 4 . At step 225, 
the corresponding images in original panels are then texture- 
mapped into articulated pyramidic panels 190^ and 200^, 
which have been distorted in accordance with the movement 
of the viewer. Note that to unambiguously texture-map onto 
panels 190 1-4 , these panels are each subdivided into two 
triangular subregions and then texture-mapped. Shown in 
FIG. 11 is image 115 seen from a location away from the 
image's ideal viewing point, using the articulated pyramidic 
panel structure of the present invention. 

Note that the above articulated pyramidic panel structure 
may also use more than two sets of pyramidic panel struc- 
tures. Instead of treating the curve road as two straight 
corridors, multiple corridors may be employed, each placed 
end-to-end and extending back from screen or panel 110. 
Likewise, each corridor represents a different portion of road 
125 in the three-dimensional space of world 105, with 
features nearer the center of the image being farther away 
from the user's viewpoint. In such a case, each set of 
articulated pyramidic panels are formed reitererately using 
the above described procedure. 

Refer ring to JIGS. 12-13, there is shown a tmrcLemb.odL- 
ment oTthe present invention which' issim ilar to that of FIG. 
5 aria in which "pyramidic panels" 145 1 , 7"r45 2 t , 145 3 ' and 
145 4 * have been now multi-segmented into sections 205 J . 4 , 
210 a _ 4 , 205 and 210 1 . 4 ', respectively, with the images in 
original panels 145 2 . 4 then texture-mapped into the corre- 
sponding translated sections of the pyramidic panel 
structure, as discussed herein below. It should be recalled 
that the pyramidic panel structure represents the three- 
dimensional mapping (x,y,z) of two-dimensional image 115 
onto image screen or panel 110 (x,y). Advantageously, the 
embodiment of FIGS. 12-13 minimizes the depth profile of 
the pyramidic panel structure along the z-axis. Unlike the 
embodiment of FIG. 5, the depth profile of this third 
embodiment does not substantially vary with changes in the 
user's viewpoint, V. In the exemplary embodiment of FIG. 
5, recall that distorting image 115 so as to move the 
vanishing point from P to P results in the pyramidic panel 
structure forming a four-sided pyramid. The base of the 
pyramid is fixed and corresponds to original screen or panel 
110, with its peak located at P and moves in concert with the 
viewer's current location, V. As the user's viewpoint moves 
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along the z-axis closer to and farther from two-dimensional 
image 115, the image's new vanishing point P' moves farther 
from and closer to the user's viewpoint, respectively. This 
latter movement causes the depth profile along the z-axis of 
the pyramidic panel structure to vary accordingly. 
Unfortunately, this variation in depth profile can undesirably 
and/or unexpectedly occlude from the user's view objects in 
the virtual world, or cause objects to occlude other features 
in the virtual world inasmuch as the corresponding images 
in the panels are distorted, as discussed above herein. 

To obviate the aforementioned problem, "pyramidic pan- 
els" 14S U4 \ 145 2 ', 145 3 ' and 145 4 have been multi- 
segmented into sections 205^, 210^, 205 1 . 4 ', and 210 U4 
respectively. Each section is then translated along the z-axis 
to a predetermined distance towards or away from the user's 
viewpoint, V, but importantly of the same orientation as the 
original section. For example, segmented sections 20S U4 
and 205 U 4 may each have one of its outer edge along the 
x-axis translated to he on the x,y plane of screen or panel 
110, as shown in phantom in FIG. 12. As the user moves to 
a new viewpoint, each section in effect pivots about that 
edge along the x-axis, which edge lies on the surface of 
panel 110. Similarly, section 210^ and 21Q 1 _ 4 1 may each 
have one of it outer edge along the y-axis lying on the 
surface of panel 110. Alternatively, sections 205 1 . 4 , and 
205 1 . 4 f may be centered along panel 110, as depicted in FIG. 
14. Likewise, sections 210 3 _ 4 and 210^' may be similarly 
translated, but for the sake of clarity are not shown in FIGS. 
12 and 14. 

Still further, each of sections 205 1 _ 4 and 205 ^ may in 
effect be rotated or pivoted about its other edge along the 
x-axis as the user moves to a new viewpoint, V, or, in 
general, about an axis parallel with an edge along the x-axis 
of the corresponding section. Again, this latter axis may, but 
does not have to, lie on the surface of panel 110. Regardless 
of the segmenting method chosen, however, translating each 
section towards or away from the user's viewpoint signifi- 
cantly reduces the depth profile of the pyramidic panel 
structure along the z-axis, such as depicted in FIG. 12 from, 
for example, T 2 to T 2 . 

In still another embodiment of the present invention, 
sections 205 2 _ 4 and 205^' may each be translated a different 
distance along the z-axis, as illustrated in FIG. 15. Although 
not shown, sections 210^ and 210 1 . 4 * may likewise be 
translated. Those skilled in the art will readily understand 
that doing so advantageously allows the user's viewpoint, V, 
to extend in front of panel 110 inasmuch as segmented 
sections corresponding to the image's center may be offset 
and located closer to the user's viewpoint, V, than the outer 
sections. 

Also, note that segmenting the pyramidic panel structure 
into a greater number of smaller sections accordingly only 
further reduces the depth profile, which asymptotically 
approaches a zero thickness. It is contemplated that the 
number of sections that the panel structure is divided into 
may be chosen empirically based on image content as well 
as the user's range of movement within the virtual world. 
Preferably, however, the panel structure is dynamically 
segmented in a reiterative manner. For example, once a user 
has chosen the maximum desired depth for the panel struc- 
ture along the z-axis to minimize occlusion, each panel is 
then reiteratively segmented into a greater number of 
smaller sections until the depth profile is reduced to the 
maximum depth profile desired. 

In accordance with the principles of the invention, . it 
should be clearly understood, however, that to maintain the 
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apparent integrity of two-dimensional image 115 when 
texture-mapping the image onto the segmented sections, 
each segmented sections 205^, 205 J . 4 ', 210 1 _ 4 , and 210 1 . 4 t 
is scaled accordingly with respect to the user's current 
5 viewpoint, V, so as to appear to be of the same size as the 
original corresponding section. This scaling or transform is 
given by: 




where S^, is the size of the original pyramidic section; S, is 
the size of the translated, segmented pyramidic panel sec- 
tion; T is distance to the original pyramidic section from the 

15 user's viewpoint, V; and T f is the distance to the translated, 
segmented pyramidic section. In other words, each 
segmented, translated section is scaled by the ratio T/T p . Of 
course, as the user moves within the world, pyramidic panels 
145^ are accordingly re-segmented, translated, and then 

20 scaled with respect to the user's new viewpoint, V. Then, the 
images in original panels 145^ are again accordingly 
texture-mapped into the corresponding translated sections 
205^, 205^4, 210 3 . 4 , and 210^ of the pyramidic panel 
structure. 

25 In the above embo di ments, distorting two-dimension al 
i mage according to the movement of the u ser's view- 
p oint results in di fferent portions of the linag e Being accord- 
in gly iJ expanaed" or "compresse d" when texture-mapping 
ima ge 115 onto the corr espondi ng pyramidic panelsjTas 

30 di scussed above herein. Also, as me use r's vi ewpoint moves 
alo ng the z-axis closer to tw o-dmiensio nal ima g e 115, the 
ima ge is ac c ordingly scaled oT"enlarged " to m ake obj ects in 
the _jmage a ppjaT closer to ^the userrNote tfia T"3oing so 
re quires scaling the x and v components ot trie objects 

35 accordingly as a tunction of~tEeir~aistan ce (z^axis )rTo^rie 
u ^er A s~YfewpoiBTTT6~til^ of Th e 

pres ent invention, shown in FIG. 16 is screen 110 with 
two -dimensional image 115 d isplayed therein Having aTeso- 
lu tion of MxN pixels, e. g. , 1200xl 2007Now,~as "shown*in 

40 " FIG. 17, m oving along the z-axis closer to the center of "the 
i mage and Th en .texture-mapping tHeT.correi p^dm g poTtions 
of image 115 onto pyramidic panels 145 1 . 4 I req uires enlarg- 
ing ancTclistortihg the ima g e such TKaTonl v„a_small.center 
p ortion of the enl arged and distorted image is displayed on 

45 scr een 110, whictPirTeffecriowers thj^bsen^able~imag e 
reso lution. This _is_so s ince the image only has a fini te 
nu mber ot pixels, and few £rpixeirare*9isprayed on the same 
si ze screen._Qne solution to tfiislatter ^robTem isTisirjg_higri 
r esolution ima g es whicOl lows"the user to travel towards 

50 the center ofjhe im age. without lo^i ngTnTa*g£ qualit5Cbut.it. 
re sults in the perimeter j)Lthe_irnage having a resolution 
highejj^it_neej,ed.since_enlarging tfie* im age'causes" the 
perimeter of the image to b e outside the field of view of the 
o bserver and, the refore not Bis played"on "the*screen7This 

55 hig her than neede_d _resolution requires more memory~to 
store image ll^as.weff.as.adfoti pnal^ 
per foTm*the_ texture.mapping, among other trJings. Of cour se, 
using lower resol ution.images^wou ld require less, memor y 
and computational time, but it typ ically leads to poor image 

60 qualit y near the center of ima ge where ttie~image is typically, 
enlarged or distorted Jhe_greatest: 

Referr ing to FIG. 18 T there is shown still another e mbod i- 
me nt ot thepresent invention which obviates the af oremen- 
tiWdjTrnblern by nsj pg a two-dimensional image Having a 

65 "hi erarchical image resolution," thereby allowing seg- 
mented portions of the image to have a resolu tion according 
to its location within the image. In this manner, a fine 
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re solution ca n be used for portions of im ag e 115 near jhc 
c enter on fitTimag e whereas a coarse resohiti on. can be used 
f or the perimeter, t hereb y minimizing the total texture-ma p 
si ze or pixel-maj^size . It sh ould be clearl y unders t ood tha t 
this hierarchical image resol ut ion is equall y applicable to 5 
any^5e~ "pfnhTabove pvrami9ic_ paneLstruct ure*s discusse d 
herein above. In this aspect of the present invention, two- 
dimensional image 115 has been segmented into resolution 
regions 215^ each having a rectangular shape, although 
other shapes are readily adaptable for the purposes of this 
invention. The corresponding portions of image 115 within 
resolution regions 215^ have different resolutions such that 
portions of the image closer to the center of the image have 
a finer resolution than portions farther away. For example, 
resolution regions 215]. 4 may have image pixel resolutions 
of m i xm 1 , m 2 xm 2 , m 3 xm 3 and m 4 xm 4 , respectively, where 15 
m 1 ^m 2 ^m 3 =m 4 . In this manner, distorting or enlarging 
portions of the image near the center of the image or where 
the image's vanishing point is typically located does not 
substantially affect the observed image resolution inasmuch 
as a greater number of pixels are contained in the image. 20 

It is contemplated that the number of resolution region s 
thaV im ag e il5 is segmented into as well as the Pixel 
r esolution therein may be, for example, chosen empiricalTy 
ba sed on Jma g e content a s well a s the user's range of 
movement within the virtual worUTFor exam ple, portions o f 25 
t he image neartne center can h ave a resolutiolTof~4809 x 
4800 ^hflcporTto~fls near the perimeter only a resolutiono f 
t2QQxl2QQ ^Furthermore, the_resolution for resolutio n 
reg ions 215 1 may be dynam icall y chosen according to th e 
a mount of distortio n or enlargement performed on t he 
im age. ITiat is, each region is allocated a finer resolutio n 
wit h a greater (dis tortion or enlargement, and vice-a-versa . 
"""Of caurse. ^arious com put er sto rage techniques may be 
used jo set the image resolution within regions ^15 1-4 . F or 
e xample, image 115, typically a st il l photo graph,, picture or 
vide o frame, may be captured wrtbjj^s£luUQn»Qf-4800x 
48 00 pixejsor greater, and then a subset of tho se.pixels,used 
to achiexe^a_dssir ed_lower reso lution. 

The foregoing merely illustrates the principles of the 
invention. It will thus be appreciated that those skilled in the 
art will be able to devise various arrangement which, 
although not explicitly describe or shown herein, embody 
the principles of the invention and are included within its 
spirit and scope. 

What is claimed is: 

1. A method for use in processing a view of a three- 
dimensional world in which a first portion of said world is 
modeled as computer graphics and a second portion of said 
world is represented by a two-dimensional image texture - 
mapped on a panel, said two-dimensional image including 
an object depicted in perspective, said image being such that 50 
features of the object closer to a predetermined point of the 
image are farther away from a user's viewpoint, comprising 
the steps of: 

segmenting the two-dimensional image into a plurality of 55 
sections; 

setting the pixel resolution of each of said plurality of 
sections according to its location within the two- 
dimensional image; 

determining a vector, C, corresponding to the direction of 60 
said perspective object in the three-dimensional world; 

projecting towards said panel the vector, C, from the 
two-dimensional image's ideal viewing point, IVP, the 
intersection of said vector, C, with the panel being 
denoted as the image's vanishing point, P; 55 

segmenting said panel into triangular regions intersecting 
at the image's vanishing point, P; 
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determining the current viewpoint, V, of the user and 
projecting a vector, T, from the image's ideal viewing 
point, IVP, to the current viewpoint, V; 

determining a new vanishing point for the two- 
dimensional image in accordance with the following 
relationship P'-P+T; 

distorting the triangular regions in the space of the three- 
dimensional world such that they intersect at the new 
vanishing point, P'; 

texture-mapping the two-dimensional image in the trian- 
gular regions onto said distorted triangular regions; and 

displaying the computer graphics along with the two- 
dimensional image. 

2. The invention as defined in claim 1 wherein the pixel 
resolution of each of said plurality of sections is set accord- 
ing to its location relative to the center of the two- 
dimensional image. 

3. The invention as defined in claim 1 wherein the pixel 
resolution of each of said plurality of sections is set such that 
sections closer to the center of the two-dimensional image 
have a higher pixel resolution than sections farther away. 

4. The invention as defined in claim 1 wherein each of 
said plurality of section is substantially rectangular in shape. 

5. The invention as defined in claim 1 wherein the 
vanishing point is located at or near the center of the 
two-dimensional image. 

6. A method for use in processing a view of a three- 
dimensional world in which a first portion of said world is 
modeled as computer graphics and a second portion of said 
world is represented by a two-dimensional image texture - 
mapped on a panel, comprising the steps of: 

determining the current viewpoint of the user, V; 

dividing the panel into triangular regions; 

distorting the triangular regions to form pyramidic panels 
such that a corresponding vanishing point, P, of a 
portion of the two-dimensional image moves as a 
function of the current viewpoint of the user; 

segmenting each of said pyramidic panels into a plurality 
of sections; 

translating each of said plurality of sections of said 
pyramidic panels towards, or away from, said current 
viewpoint of the user, V; 

segmenting the two-dimensional image into sections; 

adjusting the pixel resolution in each of said sections of 
the two-dimensional image according to its location 
relative to the center of the two-dimensional image; 

texture-mapping the two-dimensional image onto the plu- 
rality of sections of the pyramidic panels; 

scaling each of said plurality of sections of said pyramidic 
panels in accordance with the following relationship 
SfSpTJTp, where S p is the size of the section; S, is the 
size of the translated section; T^ is distance to the 
section from the user's viewpoint, V; and T f is the 
distance to the translated section from the user's 
viewpoint, V; and 

displaying the computer graphics along with the two- 
dimensional image. 

7. The invention as defined in claim 6 wherein the pixel 
resolution of each of said plurality of sections is set such that 
sections closer to the center of the two-dimensional image 
have a higher pixel resolution than sections farther away. 

8. The invention as defined in claim 6 wherein each of 
said plurality of section is substantially rectangular in shape. 

9. The invention as defined in claim 6 wherein said 
segmenting step includes resegmenting said pyramidic pan- 
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els into a greater number of smaller sections until the depth 
profile of the pyramidic panel structure formed from said 
panels reaches a predetermined level. 

10. The invention as defined in claim 6 wherein an outer 
edge of each of said plurality of sections of said pyramidic 5 
panels is located on the surface of a predetermined plane. 

11. The invention as defined in claim 10 wherein said 
predetermined plane is the panel onto which the two- 
dimensional image is texture-mapped. 

12. The invention as defined in claim 6 wherein the center 10 
of each of said plurality of sections of said pyramidic panels 

is substantially located at the surface of a predetermined 
plane. 

13. The invention as defined in claim 12 wherein said 
predetermined plane is the panel onto which the two- 15 
dimensional image is texture-mapped. 



12 



14. The invention as defined in claim 6 further comprising 
determining a vector, C, corresponding to the direction of a 
portion of a path contained within the two-dimensional 
image, and projecting toward the panel the vector, C, from 
the image's ideal viewing point, IVP, the intersection of said 
vector, C, with the panel being denoted as the image's 
vanishing point, P. 

15. The invention as defined in claim 6 wherein said 
distorting of the triangular regions in said distorting step 
includes determining a new vanishing point, P', for said 
two-dimensional image in accordance with the following 
relationship P'=P+T, wherein T is a vector from the image's 
ideal viewing point, IVP, to the current viewpoint, V. 
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(57) ABSTRACT 

A limitation of using two-dimensional images, such as 
videos or photographs, to represent portions of a three- 
dimensional world occurs when the user moves within the 
world and views the world from a location different than 
from the original context of the two-dimensional image, i.e., 
from a location different than the image's ideal viewing 
point (IVP). View changes result in the image not aligning 
well with the surrounding objects of the three-dimensional 
world. This limitation is overcome by distorting the two- 
dimensional image so as to adjust the image's vanishing 
point(s) in accordance with the movement of the user using 
a pyramidic panel structure. In this manner, as the user 
moves away from the ideal viewing point, the distortions act 
to limit the discontinuities between the two-dimensional 
image and its surroundings. To minimize the depth profile of 
the pyramidic panel structure, the structure may be seg- 
mented into sections and each section translated towards, or 
away from, the user's viewpoint. 
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DISPLAY TECHNIQUES FOR 
THREE-DIMENSIONAL VIRTUAL REALITY 

CROSS-REFERENCE TO RELATED 

APPLICATION 5 

This application is a continuation-in-part of U.S. patent 
application Ser. No. 09/107,059 filed on Jun. 30, 1998 (Case 
Edmark-2). The above-identified co-pending application, 
which is commonly assigned, is incorporated herein by 
reference. 10 

TECHNICAL FIELD 

This invention relates to the integration of three- 
dimensional computer graphics and a two-dimensional 15 
image to provide a realistic three-dimensional virtual reality 
experience. 

BACKGROUND OF THE INVENTION 

The display of a three-dimensional virtual reality world to 20 
a user requires considerable computation power, and it is 
typically cosdy to develop the necessary highly detailed 
models required for doing so. In order to simplify the 
problem, two-dimensional images, such as videos or 
photographs, may be used to represent or simulate portions 25 
of the three-dimensional world. A great reduction in com- 
putation power and cost can be achieved by such an arrange- 
ment. 

SUMMARY OF THE INVENTION 30 

A limitation of such a world occurs when a user moves 
within the world and views the world from a location 
different than the original context of a two-dimensional 
image which has been carefully calibrated to "fit into" the 
world. View changes, such as from a location different than 35 
the image's ideal viewing point, result in the image not 
aligning or fitting well with the surrounding objects of the 
three-dimensional world. We have recognized that, in accor- 
dance with the principles of the invention, viewpoint 
changes may be dealt with by distorting the two-dimensional 40 
image so as to adjust the image's vanishing point(s) in 
accordance with the movement of the user using a novel 
"pyramidic panel structure." In this manner, as the user 
moves away from the ideal viewing point, the distortions act 
to limit the discontinuities between the two-dimensional 45 
image and the surroundings of the world. In certain 
embodiments, the pyramidic panel structure may be seg- 
mented into sections, each translated towards or away from 
the user's viewpoint and then scaled, so as to minimize the 
depth profile of the pyramidic panel structure. 50 

BRIEF DESCRIPTION OF THE DRAWINGS 
In the drawings: 

FIG. 1 shows an example of that which a user sees when 55 
a user views the world from the ideal viewing point for a 
two-dimensional image representing a portion of the world; 

FIG. 2 shows an example of that which a user sees when 
a user moves within the world of FIG. 1 and views the 
two-dimensional image from a location different than the go 
image's ideal viewing point, without the use of the present 
invention; 

FIG. 3 shows an exemplary process, in accordance with 
the principles of the invention, for distorting the two- 
dimensional image using a pyramidic panel structure so as 65 
to adjust the image's vanishing point in accordance with the 
movement of the user; 
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FIGS. 4 and 5 depict the pyramidic panel structure of the 
present invention for distorting the two-dimensional image 
so as to adjust the image's vanishing point, in accordance 
with the movement of the user; 

FIGS. 6A-B depict examples of that which a user sees 
when a user views the world from a location left of the 
image's ideal viewing point, without and with the use of the 
present invention, respectively; 

FIGS. 7A-B depict examples of that which a user sees 
when a user views the world from a location above the 
image's ideal viewing point, without and with the use of the 
present invention, respectively; 

FIGS. 8A-B depict examples of that which a user sees 
when a user views the world from a location toward the front 
and the right of the image's ideal viewing point, without and 
with the use of the present invention, respectively; 

FIG. 9 shows an exemplary process, in accordance with 
the principles of the invention, for distorting a two- 
dimensional image using an articulated pyramidic panel 
structure so as to adjust multiple vanishing points in the 
image, in accordance with the movement of the user; 

FIG. 10 depicts an example of the articulated pyramidic 
panel structure of the present invention; 

FIG. 11 depicts an example of that which a user sees when 
a user views the world from a location away from the ideal 
viewing point of the two-dimensional image, with the use of 
the articulated pyramidic panel structure of the present 
invention; 

FIGS. 12 and 13 depict side and front views, respectively, 
of the pyramidic panel structure of FIG. 5 with each panel 
segmented into a plurality of sections; 

FIG. 14 depicts the pyramidic panel structure of FIG. 5, 
with each panel segmented into a plurality of sections 
having its centers located on the surface of a predetermined 
plane; and 

FIG. 15 depicts the pyramidic panel structure of FIG. 5, 
with each panel segmented into a plurality of sections and 
each section translated a different distance toward the user's 
view point, V. 

DETAILED DESCRIPTION 

To better understand the invention, FIGS. 1-2 show 
examples of that which a user sees when the user moves 
within a three-dimensional virtual reality world (x,y,z) and 
views a two-dimensional image (x,y) representing a portion 
of the world from a location at the image's ideal viewing 
point (I VP), and then from a different location, i.e., a 
location different than the original context of the image. It 
should be understood that the two-dimensional image has 
been carefully calibrated to "fit into" the surroundings of the 
world. For simplification of terminology purposes, we shall 
use the term two-dimensional image to denote either a video 
clip or a photograph. In accordance with the principles of the 
invention, as the user moves away from the ideal viewing 
point, discontinuities between the two-dimensional image 
and its surroundings are minimized by distorting the image 
according to the movement of the user. 

FIG. 1 shows an exemplary three-dimensional reality 
world 105, which is a bicycle path in a park, e.g., Central 
Park in New York City. In representing world 105, the 
present invention exploits a characteristic common for 
images consisting of views looking down the center of 
roads, streets or paths, which is that they may be treated as 
perspective, corridor-like images, with features closer to the 
center of the image being farther, away from the viewer 
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along the z-axis. Accordingly, the bicycle path or road and dimensional image 115 according to viewer's position so as 

its immediate vicinity are treated as a kind of three- to adjust the image's vanishing point with the viewer's 

dimensional, corridor-like image whose floor is formed by movement. As the viewer moves from the image's ideal 

the roadbed, whose ceiling is formed by the sky, and whose viewing point, these distortions act to limit discontinuities 

sidewalls are formed by the roadside objects. In this manner, 5 between image 115 and the surroundings of CG Part 120. 

the principles of a simple point perspective can be used for piG. 3 shows an exemplary process in accordance with 

distorting the landscape image in accordance with the move- the principles of the invention for distorting two- 

ment of the viewer, as discussed herein below. dimensional image 115 so as to adjust its vanishing point in 

World 105 is divided into two portions, screen or panel accordance with the viewer's position. The process is 

110 on which is shown or displayed a two-dimensional 10 entered at step 130 whenever it is determined that the 

image 115, such as a still photograph, picture, or a current viewer's position has changed. 

frame of a video clip; and the remainder of the world 120, Using tne v j rtua i wor ld's road model of the CG Part 105, 

which is represented using computer graphics techniques, a vector, €, corresponding to the direction of road 125 is 

and is thus referred to herein as computer graphics (CG Part) projected at step 135 from the image's ideal viewing point, 

120. Within CG Part 120 there are various synthetic, three- is IV p, to panel or screen 110 on which is displayed image 115. 

dimensional landscapes or objects modeled in, for example, N ote that the panel is two-dimensional, but represents three- 

the Virtual Reality Modeling Language (VRML). Two- dimensional space with objects nearer the center of the 

dimensional image 115 simulates landscape or terrain por- i^ge b emg farther away from the plane of the viewer. The 

tions of the world 105, here a virtual road or course 125 for pane i structure is shown in FIG. 4. The point of intersection 

walking, running or pedaling a bicycle. 20 wim screen Qr pand no fc tfae vanishing pomtj R 

Note that although three-dimensional world 105 cannot be Note, however, that the vanishing point may be set visually 

actually rendered in a two-dimensional plane (x,y), it can be by the user, if desired, or by other suitable computer graphics 

projected to and displayed on a two-dimensional plane so as processing techniques known in the art. Next, in step 140, 

to appear to have three dimensions (x,y,z). Accordingly, the screen or panel 110 is segmented into four triangular regions 

techniques of the present invention are preferably employed 25 145 1-4J one for each of the regions bordering CG Part 120, 

with computers and software, which are sufficiently sophis- with the intersection point of the four regions located at the 

ticated to display images on a two-dimensional plane as vanishing point, P. 

having three dimensions. Note also that to make the world Thereafter in step 150, the current viewpoint of the user, 

look realistic, computer graphics display techniques use the y, is determined, and a vector T projected from the ideal 

z component of objects to scale accordingly the x and y viewing point, IVP, to the viewer's current location, V. In 

components as a function of its distance (z-axis) to the user's accordance with the principles of the invention, as the 

viewpoint. viewer moves, a new vanishing point P' is calculated as 

Two-dimensional image 115 is carefully placed, cropped P'oP+T. The four triangular regions 145^ are distorted in 

and sized to achieve continuity with the surrounding envi- 35 the three-dimensional space of the virtual world at step 155 

ronment of the CG Part 120. Note that the image is clipped to represent the mapping of objects nearer the center of the 

so that the left and right edges of the road in CG Part 120 image being displaced farther away from the viewpoint of 

pass through the left and right bottom comers of the road, the user. The four triangular regions intersect at the new 

respectively, in image 115. This clipping ensures that the vanishing point P' and form so-called "pyramidic panels" 

roadbed maps to the floor of the hypothetical corridor. In so 4Q 145^.4. This is illustrated in FIG. 5. At step 160, the 

doing, portions at the boundary between two-dimensional corresponding images displayed in regions 145^4 are then 

image 115 and CG part 120 are co-planar, i.e., at the same "texture-mapped" onto pyramidic panels 145^.4, respec- 

distance away along the z-axis from the user's viewpoint. In lively. In this manner, as the viewer moves away from the 

"fitting" two-dimensional image 115 to CG part 120, image's ideal viewing point, IVP, distortions in the image 

however, there exits only one viewpoint from which that 45 resulting from moving the image's vanishing point from P to 

image's content properly corresponds to the surrounding P' act to limit the discontinuities between image 115 and the 

environment of CG Part 120. This unique location is called surroundings within CG Part 105. 

the image's ideal viewing point (IVP). In FIG. 1, two- In the exemplary illustration of FIG. 5, distorting image 
dimensional image 115 is seen from its ideal viewing point, n 5 ^ as to move the vanishing point from P to P results in 
and from this view, image 115 aligns well with the surround- 5Q pyram idic panel structure forming a four-sided pyramid, 
ing objects of CG Part 120. Note that its 5ase ^ ^ corresponds to original screen 
Users, however, rarely view image 115 only from its idea or panel 110, with its peak located at P', which moves in 
viewing point. As the user moves within world 105, such as concert with the viewer's current location, V. As the user's 
left or right of road 125, as they round curves, or move closer viewpoint moves closer to and farther from the image, the 
to or farther from the image, they see image 115 from 55 image's vanishing point accordingly moves farther from and 
positions other than its ideal viewing point. Absent the use closer to the user's viewpoint, respectively, 
of the present invention, such viewpoint changes would F IGS. 6 through 8 compare the display of two- 
cause objects or features within image 115 to align improp- dimensional image 115 on screen or panel U0 with the 
erly with the surrounding environment, as further illustrated display of the same image using the "pyramidic" panels of 
FIG ' 2 - 60 the present invention. More specifically, FIGS. 6A, 7A and 
In accordance with the principles of the invention, 8A depict viewing two-dimensional image 115 at a location 
however, screen or panel 110 uses a display structure called from the left, above, and in front and to the right of the 
a "pyramidic panel structure" for displaying two- image's ideal viewing point, IVP, respectively, without the 
dimensional image 115 within the surrounding three- use of the present invention. In these latter figures, note that 
dimensional space of the CG Part 105 so as to deal with 65 there are discontinuities between the edges of the road and 
viewpoint changes. The transformations associated with the the three-dimensional space of CG Part 105. FIGS. 6B, 7B 
pyramidic panel structure dynamically distort two- and 8C depict the same two-dimensional image distorted 
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and texture-mapped onto pyramidic panels 145' 3 _ 4 , in accor- the viewer move, the four internal points Ql through Q4 are 

dance with the principles of the invention. Note that in these mapped with the viewer's movement to Ql' through Q4', 

latter figures, the discontinuities in the road edge have been respectively, in accordance with the following relationship: 

substantially eliminated. Q/«Q t +aT, at step 220. Note that doing so, accordingly 

In another embodiment of the present invention, a modi- 5 distorts the first set of pyramidic panels 190 a _ 4 At step 225, 

, ... , . . r , j ♦ J i -*u tne corresponding images in original panels are then texture- 

fled pyramid* panel structure may be used to deal with d . n P lo ^ *, lte § ami | c P nels 190 and 2Wi 

two-dimensional images containing curved roads, streets, ^ haye ^ 

in accordance with the movement 

paths and other corridor-uke images containing mulUple Qf ^ yiewef ^ ^ tQ unambi usl texture . map onto 

ratherthanasmglevanishingpoint n to latter case, screen 19Q ^ m ^ subdivided inl0 ^ 

or panel U0 is segmented using multiple vanishing pointsto 10 ^ ^ bt ^ oas \ ad men texture-mapped. Shown in 

form a so called articulated Pyramidic pane structure. The * U5 ^ from a location aw from ^ 

transformations associated with the articulated pyramidic . , . , , r . . . . , / ... 

. ,. , r v c unage s ideal viewing point, using the articulated pyramidic 

panel structure dynamically distort different portions or °. „ c .° r . ° . 

r . . / A panel structure of the present invention, 

two-dimensional image 115 according to viewer positions so r vr . , , 

as to adjust the different vanishing points of the image with is Not f «»>« * bo ™ articulated pyramid* panel structure 

the viewer's movement. Likewise, as the viewer moves from ma y al f use , m ° re ^ tw ° of PJ™™^ P anel st ™- 

the image's ideal viewing point, these distortions act to limit tures^ Instead of treating the curve road as two straight 

the discontinuities between two-dimensional image 115 and corndors multiple corridors may be employed, each placed 

the surroundings of CG Part 120. end-to-end and extending back from screen or panel 110 
„„ „ , , . , .,20 Likewise, each corridor represents a different portion of road 

FIG. 9 shows an exemplary process in accordance with 125 m the three ^ imensional space of world 105 , with 

the principles of the invention for distorting two- fcatures ntwl |he aMa of the ; bej far(her aw 

dimensional image 115 using an aruculated pyramidic panel from the usef , s In such a cas6j each Mt of 

structure. Again, the process is entered at step 170 whenever articulated pyram i dic panels are formed reiteratively using 

it is determmed that the viewer's position has changed. In , he above described procedure. 

general, curve road 125 is treated as two straight corridors Refcrrin tQ mQS u _ n there ^ shown a thifd embodi . 

placed end-to-end, extending back from screen or panel 110. 4 *■ u- u • *i . *u . <?tti^ 

£ . ' , 5-ir . „• c j f%£ * ment of the present invention which is similar to that of FIG. 

Each corridor represents a different portion or road 125 in _ , . *. , (( ... . „ 1Ae} 1Ae , A 

. . ,. . , r ij iA ff -.u r * 5 an d in which "pyramidic panels 145',, 145',, 145' 3 and 

the three-dimensional space of world 105, with features AA . } . . * J ... r . , . , .r -2. 

t - . f , . - ^ L r 145 \ have been now multi-segmented into sections 205, 4 , 

nearer the center of the image being farther away from the * ACI . ^ 1A( *• i *u .u ™ 

, & & / 30 210 1 _ 4 , 205 V,_ 4 , and 210^, respectively, with the images m 

user vi wpo . original panels 145 1 4 then texture-mapped into the corre- 
Using the virtual world's road model of the CG Part 105, sp0 nding translated sections of the pyramidic panel 
corresponding directional vectors C x and C 2 of the corridors structure, as discussed herein below. It should be recalled 
are determined at step 175. Note that portion of the road ^at the pyramidic panel structure represents the three- 
nearer to the user's viewpoint is represented by and the 35 dimensional mapping (x,y,z) of two-dimensional image 115 
portion farther away is represented by C 2 . Next, in step 180, onto image screen or panel 110 ( x y y Advantageously, the 
using the vectors Cj and C 2 , the corresponding vanishing embodiment of FIGS. 12-13 minimizes the depth profile of 
points Pi and P 2 are determined, respectively, for each the pyramidic panel structure along the z-axis. Unlike the 
corridor by projecting those vectors from the image's ideal embodiment of FIG. 5, the depth profile of this third 
viewing point, I VP. Alternatively, vanishing points V 1 and P 2 4Q embodiment does not substantially vary with changes in the 
may be determined visually by the user, or by some other uscr > s vieW p oint7 V . In the exemplary embodiment of FIG. 
suitable means known in the art. In step 185, using the first 5> recan that distorting image 115 so as to move the 
corridor's vanishing point, P A , a first set of pyramidic panels vanishing point from P to P* results in the pyramidic panel 
190^4 are constructed to intersect at vanishing point, ? l9 as structure forming a four-sided pyramid. The base of the 
shown in FIG. 10. 45 p Vram id i s fixed and corresponds to original screen or panel 
Now at step 195, a coupling ratio a is calculated accord- 110, with its peak located at P' and moves in concert with the 
ing to the following equation: a=l/(l+d), where 1 is the viewer's current location, V. As the user's viewpoint moves 
length of the first corridor, and d is the distance between the along the z-axis closer to and farther from two-dimensional 
image's ideal view point (I VP) and the base of pyramidic image 115, the image's new vanishing point P' moves farther 
panels 190^ . Each line segment connecting a corner of the 50 from and closer to the user's viewpoint, respectively. This 
base to vanishing point P : is then divided into two segments latter movement causes the depth profile along the z-axis of 
by a point placed according to the coupling ratio, a. More the pyramidic panel structure to vary accordingly, 
specifically, the length I' of each line segment from the Unfortunately, this variation in depth profile can undesirably 
corner of the base of panels 190^4 to this point is given by and/or unexpectedly occlude from the user's view objects in 
r«cd", where 1' is the total length of the segment between the 55 the virtual world, or cause objects to occlude other features 
corner of the panel and the vanishing point, P^ These four in the virtual world inasmuch as the corresponding images 
points labeled Ql through Q4 are connected to form the base in the panels are distorted, as discussed above herein, 
of a second set of smaller pyramidic panels 200^ embedded x 0 obviate the aforementioned problem, "pyramidic pan- 
within the larger panels (step 205), as further illustrated in e is» 145^ 145^ 145< 3 an d 145' 4 have been multi-segmented 
FIG. 10. The intersection point of pyramidic panels 200,. 4 60 mt0 sections 205^, 210^, 205V 4f and 210'^ respectively, 
is then moved from Pi to vanishing point, P 2 . £ acn section is then translated along the z-axis to a prede- 
For the articulated pyramidic panel structure, the current termined distance towards or away from the user's 
viewpoint of the user, V, is determined, and a vector T viewpoint, V, but importantly of the same orientation as the 
projected from the ideal viewing point, IVP, to the viewer's original section. For example, segmented sections 205 1 . 4 
current location, V (step 210). As the viewer moves, a new 65 and 205^.4 may each have one of its outer edge along the 
vanishing point P' 2 is calculated as P 2 '-P 2 +T al step 215, and x-axis translated to lie on the x,y plane of screen or panel 
panels 200 1 . 4 are then distorted so as to intersect at P' 2 . As 110, as shown in phantom in FIG. 12. As the user moves to 
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a new viewpoint, each section in effect pivots about that 
edge along the x-axis, which edge lies on the surface of 
panel 110. Similarly, section 210 1 . 4 and 210' 1 _ 4 may each 
have one of it outer edge along the y-axis lying on the 
surface of panel 110. Alternatively, sections 205^, and 5 
205' 1 _ 4 may be centered along panel 100, as depicted in FIG. 
14. Likewise, sections 210^ and 210 , 1 _ 4 may be similarly 
translated, but for the sake of clarity are not shown in FIGS, 
12 and 14. * 

Still further, each of sections 205 1-4 and 205'^ may in 10 
effect be rotated or pivoted about its other edge along the 
x-axis as the user moves to a new viewpoint, V, or, in 
general, about an axis parallel with an edge along the x-axis 
of the corresponding section. Again, this latter axis may, but 
does not have to, lie on the surface of panel 110. Regardless is 
of the segmenting method chosen, however, translating each 
section towards or away from the user's viewpoint signifi- 
cantly reduces the depth profile of the pyramidic panel 
structure along the z-axis, such as depicted in FIG. 12 from, 
for example, T 2 to T x . 20 

In still another embodiment of the present invention, 
sections 205^ and 205^.4 may each be translated a different 
distance along the z-axis, as illustrated in FIG. 15. Although 
not shown, sections 210^ and 210 , 1 _ 4 may likewise be 
translated. Those skilled in the art will readily understand 25 
that doing so advantageously allows the user's viewpoint, V, 
to extend in front of panel 110 inasmuch as segmented 
sections corresponding to the image's center may be offset 
and located closer to the user's viewpoint, V, than the outer 
sections. 30 

Also, note that segmenting the pyramidic panel structure 
into a greater number of smaller sections accordingly only 
further reduces the depth profile, which asymptotically 
approaches a zero thickness. It is contemplated that the 
number of sections that the panel structure is divided into 35 
may be chosen empirically based on image content as well 
as the user's range of movement within the virtual world. 
Preferably, however, the panel structure is dynamically 
segmented in a reiterative manner. For example, once a user 
has chosen the maximum desired depth for the panel struc- 40 
ture along the z-axis to minimize occlusion, each panel is 
then reiteratively segmented into a greater number of 
smaller sections until the depth profile is reduced to the 
maximum depth profile desired. 

In accordance with the principles of the invention, it 45 
should be clearly understood, however, that to maintain the 
apparent integrity of two-dimensional image 115 when 
texture-mapping the image onto the segmented sections, 
each segmented sections 205^, 205V 4 , 210j_ 4 , and 210' 1 _ 4 
is scaled accordingly with respect to the user's current 50 
viewpoint, V, so as to appear to be of the same size as the 
original corresponding section. This scaling or transform is 
given by: 




where S p is the size of the original pyramidic section; S, is 
the size of the translated, segmented pyramidic panel sec- 60 
tion; T^ is distance to the original pyramidic section from the 
user's viewpoint, V; and T, is the distance to the translated, 
segmented pyramidic section. In other words, each 
segmented, translated section is scaled by the ratio TJT p . Of 
course, as the user moves within the world, pyramidic panels 65 
145'^ are accordingly re-segmented, translated, and then 
scaled with respect to the user's new viewpoint, V. Then, the 
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images in original panels 145 u4 are again accordingly 
texture-mapped into the corresponding translated sections 
205^, 205' 1 _ 4 , 210^4, and 210 , 1 _ 4 of the pyramidic panel 
structure. 

The foregoing merely illustrates the principles of the 
invention. It will thus be appreciated that those skilled in the 
art will be able to devise various arrangement which, 
although not explicitly describe or shown herein, embody 
the principles of the invention and are included within its 
spirit and scope. 

What is claimed is: 

1. A method for use in processing a view of a three- 
dimensional world in which a first portion of said world is 
modeled as computer graphics and a second portion of said 
world is represented by a two-dimensional image texture- 
mapped on a panel, comprising the steps of: 

determining the current viewpoint of the user, 'V; 

dividing the panel into triangular regions; 

distorting the triangular regions to form pyramidic panels 
such that a corresponding vanishing point, P, of a 
portion of the two-dimensional image moves as a 
function of the current viewpoint of the user; 

segmenting each of said pyramidic panels into a plurality 
of sections; 

translating each of said plurality of sections of said 
pyramidic panels towards, or away from, said current 
viewpoint of the user, V; 

texture-mapping the two-dimensional image onto the plu- 
rality of sections of the pyramidic panels; and 

as the user moves within the three-dimensional world, 
repeating the above steps so as to limit discontinuities 
between the two-dimensional image and the computer 
graphics. 

2. The invention as defined in claim 1 wherein said 
segmenting step includes resegmenting said pyramidic pan- 
els into a greater number of smaller sections until the depth 
profile of the pyramidic panel structure formed from said 
panels reaches a predetermined level. 

3. The invention as defined in claim 1 wherein an outer 
edge of each of said plurality of sections of said pyramidic 
panels is located on the surface of a predetermined plane. 

4. The invention as defined in claim 3 wherein said 
predetermined plane is the panel onto which the two- 
dimensional image texture-mapped. 

5. The invention as defined in claim 1 wherein the center 
of each of said plurality of sections of said pyramidic panels 
is substantially located at the surface of a predetermined 
plane. 

6. The invention as defined in claims 5 wherein said 
predetermined plane is the panel onto which the two- 
dimensional image is texture -mapped. 

7. The invention as defined in claim 1 further comprising 
scaring each of said plurality of sections of said pyramidic 
panels in accordance with the following relationship 




where S^ is the size of the section; S, is the size of the 
translated section; T p is distance the section from the user's 
viewpoint, V; and T, is the distance to the translated section 
from the user's viewpoint, V. 

8. The invention as Refined in claim 1 further comprising 
determining a vector, C, corresponding to the direction of a 
portion of a path contained within the two-dimensional 
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perspective image, and projecting toward the panel the 18. The invention as defined in claim 13 wherein an outer 

vector, U, from the image's^ ideal viewing point, I VP, the edge of each of said plurality of sections of said triangular 

intersection of said vector, C, with the panel being denoted regions is located on the surface of a predetermined plane, 

as the image's vanishing point, P. , 19. The invention as defined in claims 18 wherein said 

9. The invention as defined in claim 1 wherein said 5 predetermined plane is the panel onto which the two- 
distorting of the triangular regions in said distorting step dimensional image is texture-mapped, 
includes determining a new vanishing point, P, for said 2 0. The invention as defined in claim 13 wherein the 
two-dimensional image in accordance with the following center of each of said plurality of sections of said triangular 
relationship P'=P+T, wherein T is a vector from the image s T ^ QQS {& located m the SUfface of a predetermined plane . 
ideal viewing point, IVP to the current viewpoint, V. w 21 ^ inventkm ^ defined m ^ 2Q whefem ^ 

10 The invention as defined in claim 1 further compnsing determined lafle is the { QniQ which the ^ 

the step of calibrating the two-dimensional perspective *;* i * ■ * ^ j 

image as a function of the dimensions of the surrouodings d^enswnal image « texture-mapped, 

within the world invention as defined in claim 13 further compris- 

11. The invention as defined in claim 1 wherein said in § ^ng each of said plurality of sections of said trian- 
two-dimensional perspective image is a frame of a video. 15 &^ re S ions m accordance with the following relationship 

12. The invention as defined in claim 1 wherein said 
two : dimensional perspective image is a still picture. s =s — 

13. A method for use in processing a view of a three- p V 
dimensional world in which a first portion of said world is - 

modeled as computer graphics and a second portion of said 20 . , _ , . „ . , 

world is represented by a two-dimensional image texture- wher f S p 15 the of secUon J s < IS me S1Z ^ of * e 

mapped on a panel, said two-dimensional image including translated section; T p is distance to the section from the 

an object depicted in perspective, said image being such that user ' s viewpoint, V; and T r is the distance to the translated 

features of the object closer to a predetermined point of the section from the user's viewpoint, V. 

image are farther away from a user's viewpoint, comprising 2 s 23- ^ apparatus for use in processing a view of a 

the steps of: three-dimensional world in which a first portion of said 

determining a vector, C, corresponding to the direction of world is modeled as computer graphics and a second portion 

said perspective object in the three-dimensional world; of said world is represented by a two-dimensional perspec- 

projecting towards said panel the vector, C, from the tive image, said apparatus comprising: 

two-dimensional image's ideal viewing point, IVP, the 30 means for determining the current viewpoint of the user, 

intersection of said vector, C, with the panel being V; 

denoted as the image's vanishing point, P; means for dividing the panel into triangular regions; 

segmenting said panel into triangular regions intersecting ^ ^ usef mQyc& ^ lhreeKlimeasional world , 

at the image s vanishing point, P; mea ns for dynamically distorting the triangular regions 

determining the current viewpoint, V, of the user and 35 tQ form idic els such that a corresponding 

projectmg a vector, T, from the image s ideal viewing y ^ m p of a ^ of ^ two . dimensional 

point, IVP, to the current viewpoint, V: or c .\ c t , . . . . c 

. r * . . . . r . \ , image moves as a function of the current viewpoint of 

determining a new vanishing point for the two- me ^ lscr 

dimensional image in accordance with the following ' 

relationship P'=P+T' means for segmenting each of said pyramidic panels into 

distorting the triangular regions in the space of the three- 40 a P luralit y of sections; 

dimensional world such that they intersect at the new means for translating each of said plurality of sections of 

vanishing point, P; said pyramidic panels towards, or away from, said 

segmenting each of said triangular regions into a plurality current viewpoint of the user, V; and 

of sections; means for texture -mapping the two-dimensional image 

translating each of said plurality of sections of said onto the plurality of sections of the pyramidic panels, 

triangular regions towards, or away from, said current 24. The invention as defined in claim 23 wherein an outer 

viewpoint of the user, V; and edge of each of said plurality of sections of said pyramidic 

texture-mapping the two-dimensional image in the trian- panels is located on the surface of a predetermined plane, 

gular regions onto the corresponding sections of said 25. The invention as defined in claim 24 wherein said 

triangular regions. 50 predetermined plane is the panel onto which the two- 

14. The invention as defined in claim 13 wherein said dimensional image is texture-mapped. 

segmenting step of said triangular regions includes reseg- 26. The invention as defined in claim 23 wherein the 

menting said triangular regions into a greater number of center of each of said plurality of sections of said pyramidic 

smaller sections until the depth profile of the pyramidic panels fe located 0D me surface of a predetermined plane, 

panel structure formed from said triangular regions reaches 55 2? ^ inventioQ ^ defined ^ daim 23 com pris- 

a predetermmed level. ing means for scaling each of said plurality of sections of 

15. The invention as denned in claim 13 wherein said .< *j- T j ,/ , f „ . 

. . , . . ...... \ - " said pyramidic panels in accordance with the following 

predetermmed pomt is substantially near the center of the reiatnVm^hm 

two-dimensional image. 

16. The invention as defined in claim 13 further compris- 60 
ing displaying the texture-mapped two-dimensional image s, =s p j- 
merged with the first portion of said world that is modeled p 
as computer graphics. 

17. The invention as defined in claim 13 further compris- where S p is the size of the section; S, is the size of the 
ing the step of calibrating the two-dimensional image as a 65 translated section; T^ is distance to the original section from 
function of the dimensions of the surroundings within the the user's viewpoint, V; and T, is the distance to the 
world. translated section from the user's viewpoint, V. 



relationship 
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28. The invention as defined in claim 23 further compris- 
,ing means for determining a vector, C, corresponding to the 
direction of a portion of a path contained within the two- 
dimensional perspective image, and means for projecting 
toward the panel the vector, from the image's ideal 5 
viewing point, IVP, the intersection of said vector, C, with 
the panel being denoted as the image's vanishing point, P. 

29. The invention as defined in claim 23 wherein said 
means for distorting the triangular regions includes means 
for determining a new vanishing point, P', for said two- 10 
dimensional image in accordance with the following rela- 
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tionship P'-P+T, wherein T is a vector from the image's 
ideal viewing point, IVP, to the current viewpoint, V. 

30. The invention as defined in claim 23 further compris- 
ing means for calibrating the two-dimensional perspective 
image as a function of the dimensions of the surroundings 
within the world. 

31. The invention as defined in claim 23 wherein said 
two-dimensional perspective image is a frame of a video. 

32. The invention as defined in claim 23 wherein said 
two-dimensional perspective image is a still picture. 

* * * * * 
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videos or photographs, to represent portions of a three- 
dimensional world occurs when the user moves within the 
world and views the world from a location different than 
from the original context of the two-dimensional image, i.e., 
from a location different than the image's ideal viewing 
point (IVP). View changes result in the image not aligning 
well with the surrounding objects of the three-dimensional 
world. This limitation is overcome by distorting the two- 
dimensional image so as to adjust the image's vanishing 
point(s) in accordance with the movement of the user. In this 
manner, as the user moves away from the ideal viewing 
point, the distortions act to limit the discontinuities between 
the two-dimensional image and its surroundings. 
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DISTORTING A TWO-DIMENSIONAL FIGS. 7A-B depict examples of that which a user sees 

IMAGE TO REPRESENT A REALISTIC when a user views the world from a location above the 

THREE-DIMENSIONAL VIRTUAL REALITY image's ideal viewing point, without and with the use of the 

TECHNICAL FIELD 5 P reseQt invenlion > respectively; 

„_ . . . . r L FIGS. 8A-B depict examples of that which a user sees 

This invention relates to the integration of three- , . f n . . , , . _ 4 

.. t , & when a user views the world from a location toward the front 

dimensional computer graphics and a two-dimensional JlL . Ll . lL . , . . 

j v *• H j- i -^i and the right of the image s ideal viewing point, without and 

image to provide a realistic three-dimensional virtual reality ... tU f4 , . &r . . 

. r J with the use of the present invention, respectively; 
experience. 

10 FIG. 9 shows an exemplary process, in accordance with 

BACKGROUND OF THE INVENTION the principles of the invention, for distorting a two- 

The display of a three-dimensional virtual reality world to dimensional image using an articulated pyramidic panel 

a user requires considerable computation power, and it is structure so as to adjust multiple vanishing points in the 

typically costly to develop the necessary highly detailed is ima S e » in accordance with the movement of the user; 

models required for doing so. In order to simplify the FIG. 10 depicts an example of the articulated pyramidic 

problem, two-dimensional images, such as videos or panel structure of the present invention; and 

photographs, may be used to represent or simulate portions FIG. 11 depicts an example of that which a user sees when 

of the three-dimensional world. A great reduction in com- a user views the world from a location away from the ideal 

putation power and cost can be achieved by such an arrange- 20 viewing point of the two-dimensional image, with the use of 

ment. the articulated pyramidic panel structure of the present 



SUMMARY OF THE INVENTION 



invention. 



A limitation of such a world occurs when a user moves 25 DETAILED DESCRIPTION 

within the world and views the world from a location To better understand the invention, FIGS. 1-2 show 
different than the original context of a two-dimensional examples of that which a user sees when the user moves 
image carefully calibrated to "fit into" the world. View ^thin a three-dimensional virtual reality world and views a 
changes, such as from a location different than the image's two-dimensional image representing a portion of the world 
ideal viewing point, result in the image not aligning or fitting 30 fr 0m a location at the image's ideal viewing point (I VP), and 
well with the surrounding objects of the three-dimensional then from a different location, i.e., a location different than 
world. We have recognized that, in accordance with the the original context of the image. It should be understood 
principles of the invention, viewpoint changes may be dealt tnat the two-dimensional image has been carefully calibrated 
with by distorting the two-dimensional image so as to adjust 35 t0 "fit into" the surroundings of the world. For simplification 
the image's vanishing point(s) in accordance with the move- 0 f terminology purposes, we shall use the term two- 
ment of the user. In this manner, as the user moves away dimensional image to denote either a video clip or a pho- 
from the ideal viewing point, the distortions act to limit the tograph. In accordance with the principles of the invention, 
discontinuities between the two-dimensional image and the as me user moves away from the ideal viewing point, 
surroundings of the world. 40 discontinuities between the two-dimensional image and its 

BRIEF DESCRIPTION OF THE DRAWINGS surroundings are minimized by distorting the image accord- 



In the drawings: 



ing to the movement of the user. 

FIG. 1 shows an exemplary three-dimensional reality 

FIG. 1 shows an example of that which a user sees when 45 WOfld 105 whicfa ^ a bicyde palh [n a ^ CeQtral 

a user views the world from the ideal viewing point for a Pafk ^ New Yofk Q Iq representing world 105) the 

two-dimensional image representing a portion of the world; pfesent invention exploits a characteristic common t0 

FIG. 2 shows an example of that which a user sees when images consisting of views looking down the center of 

a user moves within the world of FIG. 1 and views the roadSj streets or pathS) which ^ that they may 5e trealed ^ 

two-dimensional image from a location different than the 50 perspectivej corridor-like images, with features closer to the 

image's ideal viewing point, without the use of the present center of the image being farther away from the viewer, 

invention; Accordingly, the bicycle path or road and its immediate 

FIG. 3 shows an exemplary process, in accordance with vicinity are treated as a kind of three-dimensional, corridor- 

the principles of the invention, for distorting the two- 55 like image whose floor is formed by the roadbed, whose 

dimensional image using a pyramidic panel structure so as ceiling is formed by the sky, and whose side walls are formed 

to adjust the image's vanishing point in accordance with the by the roadside objects. In this manner, the principles of a 

movement of the user; simple point perspective can be used for distorting the 

FIGS. 4 and 5 depict the pyramidic panel structure of the landscape image in accordance with the movement of the 

present invention for distorting the two-dimensional image 60 viewer, as discussed hereinbelow. 

so as to adjust the image's vanishing point, in accordance World 105 is divided into two portions, screen or panel 

with the movement of the user; HO on which is shown or displayed a two-dimensional 

FIGS. 6A-B depict examples of that which a user sees image 115, such as a still photograph, picture, or a current 

when a user views the world from a location left of the 65 frame of a video clip; and the remainder of the world 120, 

image's ideal viewing point, without and with the use of the which is represented using computer graphics techniques, 

present invention, respectively; and is thus referred to herein as computer graphics (CG Part) 
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120. Within CG Part 120 there are various synthetic, three- Using the virtual world's road model of the CG Part 105, 

dimensional landscapes or objects modeled in, for example, a vector, corresponding to the direction of road 125 is 

the Virtual Reality Modeling Language (VRML). Two- projected at step 135 from the image's ideal viewing point, 

dimensional image 115 simulates landscape or terrain por- I VP, to panel or screen 110 on which is displayed image 115. 

tions of the world 105, here a virtual road or course 125 for 5 Note that the panel is two-dimensional, but represents three - 

walking, running or pedaling a bicycle. dimensional space with objects nearer the center of the 

Note that although three-dimensional world 105 cannot be image being farther away from the plane of the viewer. The 

actually rendered in a two-dimensional plane, it can be panel structure is shown in FIG. 4. The point of intersection 

projected to and displayed on a two-dimensional plane so as with screen or panel 110 is the image's vanishing point, P. 

to appear to have three dimensions. Accordingly, the tech- Note, however, that the vanishing point may be set visually 

niques of the present invention are preferably employed with by the user, if desired, or by other suitable computer graphics 

computers and software, which are sufficiently sophisticated processing techniques known in the art. Next, in step 140, 

to display images on a two-dimensional plane as having screen or panel 110 is segmented into four triangular regions 

three dimensions. Note also that to make the world look 15 145^, one for each of the regions bordering CG Part 120, 

realistic, computer graphics display techniques use the z w f tD tne intersection point of the four regions located at the 

component of objects to scale accordingly the x and y vanishing point, P. 

components as a function of its distance to the user's Thereafter in'step 150, the current viewpoint of the user, 

viewpoint. - s determined, and a vector T projected from the ideal 

Two-d,mens,onal image 115 is carefully placed, cropped 20 viewing lyp ^ ^ y fa 

and sized to achieve continuity with the surrounding env- accordance ^ the principles of me invention , ^ the 

ronment of the CG Part 120. Note that the image is clipped . . 4 „. . , , , 

. . r r , , . „ viewer moves, a new vanishing point P' is calculated as 

so that the left and nght edges of the road in CG Part 120 _ = ^ / i : r 

*u u *u i & j • u« u c *u a P =P+T. The four triangular regions 145, .4 are distorted m 

pass through the left and right bottom comers of the road, , , b 

respectively, in image 115. This clipping ensures that the 25 ^ three-dimensional space of the virtual world at step 155 

roadbed maps to the floor of the hypothetical corridor. In so to «=P'esent the mapping of objects nearer the center of the 

doing, portions at the boundary between two-dimensional ™ a g e bem 8 displaced farther away from the viewpoint of 

image 115 and CG part 120 are co-planar, i.e., at the same me user - ^ four triangular regions intersect at the new 

distance away from the user's viewpoint. In "fitting" two- vanishing point P and form so^alled "pyramidic panels" 

dimensional image 115 to CG part 120, however, there exits 1W U4 . This is illustrated in FIG. 5. At step 160, the 

only one viewpoint from which that image's content prop- corresponding images displayed in regions 145^ are then 

erly corresponds to the surrounding environment of CG Part "texture-mapped" onto pyramidic panels 145\. 49 respec- 

120. This unique location is called the image's ideal viewing lively. In this manner, as the viewer moves away from the 

point (I VP). In FIG. 1, two-dimensional image 115 is seen 35 image's ideal viewing point, I VP, distortions in the image 

from its ideal viewing point, and from this view, image 115 resulting from moving the image's vanishing point from P to 

aligns well with the surrounding objects of CG Part 120. P' act to limit the discontinuities between image 115 and the 

Users, however, rarely view image 115 only from its idea surroundings within CG Part 105. 

viewing point. As the user moves within world 105, such as In the exemplary illustration of FIG. 5, distorting image 

left or right of road 125, as they round curves, or move closer 40 115 so as to move the vanishing point from P to P' results in 

to or farther from the image, they see image 115 from pyramidic panel structure forming a four-sided pyramid, 

positions other than its ideal viewing point. Absent the use Note that its base is fixed and corresponds to original screen 

of the present invention, such viewpoint changes would or panel 110, with its peak located at P', which moves in 

cause objects or features within image 115 to align improp- 45 concert with the viewer's current location, V. As the user's 

erly with the surrounding environment, as further illustrated viewpoint moves closer to and farther from the image, the 

in FIG. 2. image's vanishing point accordingly moves farther from and 

In accordance with ' the principles of the invention, closer to the user's viewpoint, respectively, 

however, screen or panel 110 uses a display structure called FIGS. 6 through 8 compare the display of two- 

a "pyramidic panel structure" for displaying two- 50 dimensional image 115 on screen or panel 110 with the 

dimensional image 115 within the surrounding three- display of the same image using the "pyramidic" panels of 

dimensional space of the CG Part 105 so as to deal with the present invention. More specifically, FIGS. 6 A, 7A and 

viewpoint changes. The transformations associated with the 8A depict viewing two-dimensional image 115 at a location 

pyramidic panel structure dynamically distort two- 55 from the left, above, and in front and to the right of the 

dimensional image 115 according to viewer's position so as image's ideal viewing point, I VP, respectively, without the 

to adjust the image's vanishing point with the viewer's use of the present invention. In these latter Figures, note that 

movement. As the viewer moves from the image's ideal there are discontinuities between the edges of the road and 

viewing point, these distortions act to limit discontinuities the three-dimensional space of CG Part 105. FIGS. 6B, 7B 

between image 115 and the surroundings of CG Part 120. 60 and 8C depict the same two-dimensional image distorted 

FIG. 3 shows an exemplary process in accordance with and texture-mapped onto pyramidic panels 145' 1 _4» in accor- 

the principles of the invention for distorting two- dance with the principles of the invention. Note that in these 

dimensional image 115 so as to adjust its vanishing point in latter figures, the discontinuities in the road edge have been 

accordance with the viewer's position. The process is 65 substantially eliminated. 

entered at step 130 whenever it is determined that the In another embodiment of the present invention, a modi- 
viewer's position has changed. fied pyramidic panel structure may be used to deal with 
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two-dimensional images containing curved roads, streets, 
paths and other corridor- like images containing multiple 
rather than a single vanishing point. In this latter case, screen 
or panel 110 is segmented using multiple vanishing points to 
form a so called "articulated pyramidic panel structure." The 5 
transformations associated with the articulated pyramidic 
panel structure dynamically distort different portions of 
two-dimensional image 115 according to viewer positions so 
as to adjust the different vanishing points of the image with 
the viewer's movement. Likewise, as the viewer moves from 
the image's ideal viewing point, these distortions act to limit 
the discontinuities between two-dimensional image 115 and 
the surroundings of CG Part 120. 

FIG. 9 shows an exemplary process in accordance with 15 
the principles of the invention for distorting two- 
dimensional image 115 using an articulated pyramidic panel 
structure. Again, the process is entered at step 170 whenever 
it is determined that the viewer's position has changed. In 
general, curve road 125 is treated as two straight corridors 
placed end-to-end, extending back from screen or panel 110. 
Each corridor represents a different portion of road 125 in 
the three-dimensional space of world 105, with features 
nearer the center of the image being farther away from the 25 
user's viewpoint. 

Using the virtual world's road model of the CG Part 105, 
corresponding directional vectors C 1 and C 2 of the corridors 
are determined at step 175. Note that portion of the road 
nearer to the user's viewpoint is represented by C x , and the 
portion farther away is represented by C 2 . Next, in step 180, 
using the vectors C 1 and C 2 , the corresponding vanishing 
points P a and P 2 are determined, respectively, for each 
corridor by projecting those vectors from the image's ideal 35 
viewing point, I VP. Alternatively, vanishing points P a and P 2 
may be determined visually by the user, or by some other 
suitable means known in the art. In step 185, using the first 
corridor's vanishing point, P lf a first set of pyramidic panels 
190j_ 4 are constructed to intersect at vanishing point, P a , as 40 
shown in FIG. 10. 

Now at step 195, a coupling ratio a is calculated according 
to the following equation: a=l(l+d), where 1 is the length of 
the first corridor, and d is the distance between the image's 45 
ideal view point (IVP) and the base of pyramidic panels 
190 1 . 4 . Each line segment, connecting a corner of the base to 
vanishing point P 1 is then divided into two segments by a 
point placed according to the coupling ratio, a. More 
specifically, the length 1' of each line segment from the 50 
corner of the base of panels 190 1 . 4 to this point is given by 
NaT, where 1" is the total length of the segment between the 
corner of the panel and the vanishing point, P r These four 
points labeled Ql through Q4 are connected to form the base 55 
of a second set of smaller pyramidic panels 200 a ^ embedded 
within the larger panels (step 205), as further illustrated in 
FIG. 10. The intersection point of pyramidic panels 200^ 
is then moved from P 1 to vanishing point, P 2 . 

For the articulated pyramidic panel structure, the current 60 
viewpoint of the user, V, is determined, and a vector T 
projected from the ideal viewing point, IVP, to the viewer's 
current location, V (step 210). As the viewer moves, a new 
vanishing point P' 2 is calculated as P' 2 =P 2 +T at step 215, and 65 
panels 200 1 . 4 are then distorted so as to intersect at P 2 . As 
the viewer move, the four internal points Ql through Q4 are 
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mapped with the viewer's movement to Ql' through Q4 f , 
respectively, in accordance with the following relationship: 
Q'.=Q.+aT, at step 220. Note that doing so, accordingly 
distorts the first set of pyramidic panels 190^. At step 225, 
the corresponding images in original panels are then texture- 
mapped into articulated pyramidic panels 190 3 . 4 and 200 a _ 4 , 
which have been distorted in accordance with the movement 
of the viewer. Note that to unambiguously texture-map onto 
panels 190 a _ 4 , these panels are each subdivided into two 
triangular subregions and then texture-mapped. Shown in 
FIG. 11 is image 115 seen from a location away from the 
image's ideal viewing point, using the articulated pyramidic 
panel structure of the present invention. 

Note that the above articulated pyramidic panel structure 
may also use more than two sets of pyramidic panel struc- 
tures. Instead of treating the curve road as two straight 
corridors, multiple corridors may be employed, each placed 
end-to-end and extending back from screen or panel 110. 
Likewise, each corridor represents a different portion of road 
125 in the three-dimensional space of world 105, with 
features nearer the center of the image being farther away 
from the user's viewpoint. In such a case, each set of 
articulated pyramidic panels are formed reitererately using 
the above described procedure. 

The foregoing merely illustrates the principles of the 
invention. It will thus be appreciated that those skilled in the 
art will be able to devise various arrangement which, 
although not explicitly describe or shown herein, embody 
the principles of the invention and are included within its 
spirit and scope. 

What is claimed is: 

1. A method for use in processing a view of a three- 
dimensional world in which a first portion of said world is 
modeled as computer graphics and a second non-computer 
modeled portion of said world is represented by a two- 
dimensional perspective image, comprising the steps of: 

determining the current user's viewpoint of the two- 
dimensional perspective image, said image being such 
that features closer to a predetermined point of the 
image are farther away from the user's viewpoint so as 
to give a portion of the two-dimensional perspective 
image a vanishing point; 

distorting the two-dimensional perspective image so as to 
move the vanishing point of said portion of the two- 
dimensional perspective image according to the current 
user's viewpoint; and 

as the user moves within the three-dimensional world, 
repeating the above steps so as to limit discontinuities 
between the two-dimensional perspective image and 
the computer graphics. 

2. A method for use in processing a view of a three- 
dimensional world in which a first portion of said world is 
modeled as computer graphics and a second non-computer 
modeled portion of said world is represented by a two- 
dimensional perspective image, comprising the steps of: 

determining the current user's viewpoint of the two- 
dimensional perspective image, said image being such 
that features closer to a predetermined point of the 
image are farther away from the user's viewpoint so as 
to give a portion of the two-dimensional perspective 
image a vanishing point; and 

distorting the portions of the two-dimensional perspective 
image so as to adjust the corresponding vanishing point 
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of each of said of said portion according to the current 
user's viewpoint. 

3. The invention as defined in claim 1 further comprising 
displaying the distorted two-dimensional perspective image 
merged with the first portion of said world that is modeled 
as computer graphics. 

4. The invention as defined in claim 1 wherein said 
two-dimensional perspective image is a frame of a video. 

5. The invention as defined in claim 1 wherein said 
two-dimensional perspective image is a still picture. 

6. The invention as defined in claim 1 further comprising 
the step of calibrating the two-dimensional perspective 
image as a function of the dimensions of the surroundings 
within the world. 

7. A method for use in processing a view of a three- 
dimensional world in which a first portion of said world is 
modeled as computer graphics and a second non-computer 
modeled portion of said world is represented by a two- 
dimensional perspective image, comprising the steps of: 

determining the current user's viewpoint of the two- 
dimensional perspective image, said image being such 
that features closer to a predetermined point of the 
image are farther away from the user's viewpoint so as 
to give different portions of the two-dimensional per- 
spective image vanishing points; 

distorting different portions of the two-dimensional per- 
spective image so as to adjust the corresponding van- 
ishing point of each of said portions according to the 
current user's viewpoints and 

as the user moves within the three-dimensional world, 
repeating the above steps so as to limit discontinuities 
between the two-dimensional perspective image and 
the computer graphics. 

8. The invention as defined in claim 7 wherein said 
predetermined point is substantially near the center of the 
two-dimensional image. 

9. The invention as defined in claim 7 further comprising 
displaying the distorted two-dimensional perspective image 
merged with the first portion of said world that is modeled 
as computer graphics. 

10. The invention as defined in claim 7 wherein said 
two-dimensional perspective image is a frame of a video. 

11. The invention as defined in claim 7 wherein said 
two-dimensional image perspective is a still picture. 

12. The invention as defined in claim 7 further comprising 
the step of calibrating the two-dimensional perspective 
image as a function of the dimensions of the surroundings 
within the world. 

13. A method for use in processing a view of a three- 
dimensional world in which a first portion of said world is 
modeled as computer graphics and a second non-computer 
modeled portion of said world is represented by a two- 
dimensional image, comprising the step of: 

distorting the two-dimensional image, in which features 
closer to a predetermined point of the image are farther 
away from a user's viewpoint, so as to adjust a corre- 
sponding vanishing point of a portion of said two- 
dimensional image as a function of the current view- 
point of a user; and as the user moves within the 
three-dimensional world, repeating the above step so as 
to limit discontinuities between the two-dimensional 
image and the computer graphics. 

14. The invention as defined in claim 13 wherein said 
predetermined point is substantially near the center of the 
two-dimensional image. 



'9,548 Bl 

8 

15. A method for use in processing a view of a three- 
dimensional world in which a first portion of said world is 
modeled as computer graphics and a second non-computer 
modeled portion of said world is represented by a two- 

5 dimensional image texture-mapped on a panel, comprising 
the steps of: 

determining the current viewpoint of the user, V; * 
dividing the panel into triangular regions; 
10 distorting the triangular regions to form pyramidic panels 
such that a corresponding vanishing point, P, of a 
portion of the two-dimensional image moves as a 
function of the current viewpoint of the user; 
texture-mapping the two-dimensional image onto the 
15 pyramidic panels; and 

as the user moves within the three-dimensional world, 
repeating the above steps so as to limit discontinuities 
between the two-dimensional image and the computer 
20 graphics. 

16. The invention as defined in claim 15 further compris- 
ing determining a vector, corresponding to the direction 
of a portion of a path contained within the two-dimensional 
perspective image, and projecting toward the panel the 

25 vector, C, from the image's ideal viewing point, I VP, the 
intersection of said vector, U, with the panel being denoted 
as the image's vanishing point, P. 

17. The invention as defined in claim 15 wherein said 
distorting of the triangular regions in said distorting step 
includes determining a new vanishing point, P', for said 
two-dimensional image in accordance with the following 
relationship P'«P+T, wherein T is a vector from the image's 
ideal viewing point, IVP, to the current viewpoint, V. 

35 18. The invention as defined in claim 15 further compris- 
ing the step of calibrating the two-dimensional perspective 
image as a function of the dimensions of the surroundings 
within the world. 

19. A method for use in processing a view of a three- 
40 dimensional world in which a first portion of said world is 
modeled as computer graphics and a second non-computer 
modeled portion of said world is represented by a two- 
dimensional image texture-mapped on a panel comprising 
45 the steps of: 

determining the current viewpoint of the user; 
dividing the panel into different regions corresponding to 
different portions of the two-dimensional image, said 
image being such that features closer to a predeter- 
50 mined point of the image are farther away from the 
user's viewpoint so as to give different portions of the 
two-dimensional image different vanishing points; 
distorting the different regions of the panel to form 
55 articulated pyramidic panels such that the correspond- 
ing vanishing points of different portions of the two- 
dimensional image move as a function of the current 
viewpoint of the user; 
texture-mapping the two-dimensional image onto the 
60 articulated pyramidic panels; and 

as the user moves within the three-dimensional world, 
repeating the above step so as to limit discontinuities 
between the two-dimensional image and the computer 
graphics. 

65 20. The invention as defined in claim 19 wherein said 
predetermined point is substantially near the center of the 
two-dimensional image. 
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21. The invention as defined in claim 19 wherein said 
two-dimensional perspective image is a frame of a video. 

22. The invention as defined in claim 19 wherein said 
two-dimensional perspective image is a still picture. 

23. The invention as defined in claim 19 further compris- 5 
ing the step of calibrating the two-dimensional perspective 
image as a function of the dimensions of the surroundings 
within the world. 

24. The invention as defined in claim 19 wherein said first 
and second portions of the two-dimensional image include 10 
portions of a road with dissimilar directions. 

25. A method for use in processing a view of a three- 
dimensional world in which a first portion of said world is 
modeled as computer graphics and a second non-computer 15 
modeled portion of said world is represented by a two- 
dimensional image texture-mapped on a panel, said two- 
dimensional image including an object depicted in 
perspective, said image being such that features of the object 
closer to a predetermined point of the image are farther away 20 
from a user's viewpoint, comprising the steps of: 

determining a vector, C, corresponding to the direction of 
said perspective object in the three-dimensional world; 

projecting towards said panel the vector,C, from the 
, two-dimensional image's ideal viewing point, IVP, the 25 
intersection of said vector,C, with the panel being 
denoted as the image* s vanishing point, P; 

segmenting said panel into triangular regions intersecting 
at the image's vanishing point, P; 

determining the current viewpoint, V, of the user and 
projecting a vector, T, from the image's ideal viewing 
point, IVP, to the current viewpoint, V; 

determining a new vanishing point for the two- 
dimensional image_in accordance with the following 35 
relationship P'«P+T; 

distorting the triangular regions in the space of the three- 
dimensional world such that they intersect at the new 
vanishing point, P's and 

texture-mapping the two-dimensional image in the trian- 40 
gular regions onto said distorted triangular regions. 

26. The invention as defined in claim 25 wherein said 
predetermined point is substantially near the center of the 
two-dimensional image. 

27. The invention as defined in claim 25 further compris- 45 
ing displaying the texture-mapped two-dimensional image 
merged with the first portion of said world that is modeled 

as computer graphics. 

28. The invention as defined in claim 25 further compris- 5Q 
ing the step of calibrating the two-dimensional image as a 
function of the dimensions of the surroundings within the 
world. 

29. A method for use in processing a view of a three- 
dimensional world in which a first portion of said world is 55 
modeled as computer graphics and a second non-computer 
modeled portion of said world is represented by a two- 
dimensional image texture -mapped on a panel, said two- 
dimensional image including a path depicted in perspective 
wherein features of the path closer to a predetermined point 
of the image are farther away from a user's viewpoint, 
comprising the steps of: 

dividing said path into first and second portions; 
determining vectors and ^ corresponding to the 65 

direction and length of said first and second portions, 

respectively; 



60 



projecting towards said panel the vectors Q and from 
the two-dimensional image's ideal viewing point, IVP, 
the intersection of said vectors C x and with the panel 
being denoted as the image's vanishing points, ? 1 and 
P 2 , respectively; 

segmenting said panel into a first set of four triangular 
regions intersecting at the image's vanishing point, P 2 ; 

determining a coupling ratio a given by a«l/(l+d), where 
1 is the length of the vector and d is the distance 
between the image's ideal viewing point, IVP, and the 
panel; 

dividing each of the four fine segments of the first set of 
four triangular regions connecting to a corner of the 
panel and labeling those points Ql through Q4 with the 
exterior forming a first set of articulated pyramidic 
panels; 

segmenting the regions interior to the points Ql through 
Q4 into a second set of four triangular regions inter- 
secting at vanishing point P 2 ; 

moving the intersection point of the first set of the four 
triangular regions from P x to P 2 ; 

determining the current viewpoint, V, of the user and 
projecting a vector, T, from the image's ideal viewing 
point, IVP, to the current viewpoint, V; 

determining a new vanishing point for the second portion 
of the two-dimensional image in accordance with the 
following relationships, 

distorting the second set of four triangular regions in the 
three-dimensional space of the world such that they 
intersect at the new vanishing point, P' 2 thereby form- 
ing a second set of articulated pyramidic panels; 

mapping the points Ql through Q4 to Ql' through Q4 1 , 
respectively, according with the following relationship, 

Q\._ Gi .+aT, where I an integer between 1 through 4; and 

texture-mapping the corresponding first and second por- 
tions of the two-dimensional image in the first and 
second sets of triangular regions onto the first and 
second sets of articulated pyramidic panels, respec- 
tively. 

30. The invention as defined in claim 29 wherein said 
predetermined point is substantially near the center of the 
two-dimensional image. 

31. The invention as defined in claim 29 further compris- 
ing displaying the texture-mapped two-dimensional image 
merged with the first portion of said world that is modeled 
as computer graphics. 

32. The invention as defined in claim 29 further compris- 
ing the step of calibrating the two-dimensional image as a 
function of the dimensions of the surroundings within the 
world. 

33. An apparatus for use in processing a view of a 
three-dimensional world in which a first portion of said 
world is modeled as computer graphics and a second non- 
computer modeled portion of said world is represented by a 
two-dimensional perspective image, said apparatus compris- 
ing: 

means for determining the current user's viewpoint of the 
two-dimensional perspective image, said perspective 
image being such that features of a perspective object 
closer to a predetermined point of the image are farther 
away from a user's viewpoint so as to give the per- 
spective object a vanishing point; and 
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as the user moves within the three-dimensional world, dimensional perspective image being such that features 

means for repeatingly distorting the two-dimensional closer to a predetermined point of the image are farther 

perspective image so as to adjust the corresponding away from a user's viewpoint so as to give different 

vanishing point of the perspective object within said portions of the two-dimensional perspective image 

image according to the current user's view point. 5 each a vanishing point; and 

34. THe invention as defined in claim 33 wherein said ag ^ usef mQves within the ^ dimeQsional world> 
predetermined point is substantially near the center of the meaflS for repeatingly distorting portions of the ^ 
two-dimensional image. dimensional perspective image so as to adjust the 

35. The invention as defined m claim 33 further compris- corresponding vanishing points of different portions of 
ing means for displaying the distorted two-dimensional the two-dimensional image according to the current 
perspective image with the first portion of said world that is user's viewpoint. 

modeled as computer graphics. 40. The invention as defined in claim 39 wherein said 

36. The invention as defined in claim 33 wherein said predetermined point is substantially near the center of the 
two-dimensional perspective image is a frame of video. 15 two-dimensional image. 

37. The invention as defined in claim 33 wherein said 41. The invention as defined in claim 39 further compris- 
two-dimensional perspective image is still a picture. mg me ans for displaying the distorted two-dimensional 

38. The invention as defined in claim 33 further compris- perspective image merged with the first portion of said world 
ing means for calibrating the two-dimensional perspective mat is modeled as computer graphics. 

image as a function of the dimensions of the surroundings 20 42. The invention as defined in claim 39 wherein said 

within the world. two-dimensional perspective image is a frame of a video. 

39. An apparatus for use in processing a view of a 43. The invention as defined in claim 39 wherein said 
three-dimensional world in which a first portion of said two-dimensional perspective image is a still picture, 
world is modeled as computer graphics and a second non- 25 44. The invention as defined in claim 39 further compris- 
computer modeled portion of said world is represented by a mg mean s for calibrating the two-dimensional perspective 
two-dimensional perspective image, said apparatus compris- image as a function of the dimensions of the surroundings 
in §; within the world. 

means for determining the user's current viewpoint of the 

two-dimensional perspective image, said two- ***** 
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(57) ABSTRACT 

A method and apparatus are disclosed for monitoring the 
location of one or more viewers) and dynamically adjusting 
the image to compensate for the current location of the 
viewer(s). The image is adjusted to compensate for a view- 
ing location (pan angle, 0, tilt angle, <£, or distance, d) 
outside of a specified range of values. The input image is 
adjusted so that the output image appears as originally 
intended, for the current viewing location of the viewer. A 
linear transformation technique is applied to the original 
image to generate a modified image. The linear transforma- 
tion maps the pixels in the original image to a new space that 
distorts the image, such that when the modified image is 
viewed from an ofifeet viewing location the image appears as 
if being viewed from a direct viewing location. 
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METHOD AND APPARATUS FOR ADJUSTING AN 
IMAGE TO COMPENSATE FOR AN OFFSET 
POSITION OF A USER 

FIELD OF THE INVENTION 

[0001] The present invention relates to methods and appa- 
ratus for controlling a display, and more particularly, to a 
method and apparatus for automatically adjusting an image 
to compensate for an offset viewing location of a user. 

BACKGROUND OF THE INVENTION 

[0002] The consumer marketplace offers a wide variety of 
devices for displaying images, such as televisions, portable 
DVD players and computer monitors. Most advances in 
display technology have been directed to techniques for 
reducing glare and reflection resulting from light sources 
and objects located in the vicinity of the display. Typically, 
image quality is improved by employing polarizing panels 
or screen coatings (or both) to reduce glare and reflections. 

[0003] FIG. 1A illustrates a display 100 that is observed 
by a viewer 110 from a viewing location defined by a pan 
angle, 0, tilt angle, <J> (not shown), and distance, d, relative 
to the display 100. Display devices are typically optimized 
for direct viewing by the viewer from a specified viewing 
distance. In the example of FIG. 1A, the viewer 110 is 
observing the display 100 from a distance, d, with a direct 
viewing angle where the pan and tilt angles, © and <I>, are 
approximately zero degrees. 

[0004] If the viewer 110 is observing the display 100 from 
• a pan angle, 0, or tilt angle, O, (or both) that is offset from 
the intended direct viewing angle of the display 100, as 
shown in FIG. IB, then the image will appear distorted to 
the viewer 110. Generally, if the viewer 110 is observing the 
display 100 from a pan angle, 0, or tilt angle, O, (or both) 
that is oflset from a predefined viewing angle of the display 
then the portions of the displayed image appearing on the 
opposite side of the image relative to the viewing location 
will appear smaller than when viewed from the intended 
direct viewing angle. 

[0005] Similarly, if the viewer 110 is observing the display 
100 from a distance, d, that is outside of the optimized 
viewing range of the display, then the image will likewise 
appear distorted to the viewer 110. Generally, if the viewer 
110 is observing the display from a distance, d, beyond the 
optimized viewing range of the display 100, then the image 
will appear smaller to the viewer 110 than when viewed 
from the intended viewing range. It is further noted that as 
the size of the display area increases, the distortion caused 
by viewing the image from an oflset position is more 
significant. 

[0006] A need therefore exists for a method and apparatus 
for adjusting an image to compensate for an offset position 
of a viewer. A further need exists for a method and apparatus 
for adjusting an image to compensate for a viewing distance 
that is outside of an optimized viewing range of a display. 

SUMMARY OF THE INVENTION 

[0007] Generally, a method and apparatus are disclosed 
for monitoring the location of one or more viewers) and 
dynamically adjusting the image to compensate for the 
current location of the viewer(s). In particular, the image is 



adjusted to compensate for a viewing location (pan angle, 0, 
tilt angle, <£, or distance, d) outside of a specified range of 
values. The present invention employs image processing 
techniques to adjust the input image so that the output image 
appears as originally intended, for the current viewing 
location of the viewer. 

[0008] According to one aspect of the invention, the 
disclosed viewer-location image compensation system mor- 
phs an image to compensate for an offset pan angle, ©, or tilt 
angle, <E>, (or both) to compress portions of the image nearest 
the viewer and enlarge portions of the image further from the 
viewer. Likewise, the disclosed viewer-location image com- 
pensation system scales an image to compensate for a 
viewing distance, d, outside of an optimized viewing range 
of a display (d<d min or d>d ma J. 

[0009] In order to compensate for an offset viewing loca- 
tion, the original image can be adjusted using a linear 
transformation technique to generate a modified image. 
Generally, the linear transformation maps the pixels in the 
original image to a new space that distorts the image, such 
that when the modified image is viewed from an offset 
viewing location the image appears as if being viewed from 
a direct viewing location. 

[0010] A more complete understanding of the present 
invention, as well as further features and advantages of the 
present invention, will be obtained by reference to the 
following detailed description and drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0011] FIG. 1A is a top view illustrating a viewer observ- 
ing a display from a direct viewing angle; 

[0012] FIG. IB is a top view illustrating a viewer observ- 
ing a display from an oflset viewing angle; 

[0013] FIG. 2 is a schematic block diagram of a viewer- 
location image compensation system in accordance with the 
present invention; and 

[0014] FIG. 3 is a flow chart describing an exemplary 
image adjustment process embodying principles of the 
present invention. 

DETAILED DESCRIPTION 

[0015] FIG. 1 illustrates a viewer-location image compen- 
sation system 200 in accordance with the present invention. 
As shown in FIG. 2, the viewer-location image compensa- 
tion system 200 includes one or more cameras 250-1 
through 250-N (hereinafter, collectively referred to as cam- 
eras 250) that are focused on one or more viewer(s) 240 of 
a display 230. The images generated by the cameras 250 are 
utilized to derive the viewing location of a viewer 240 (pan 
angle, 0, tilt angle, O, and distance, d). The display 230 is 
any type of image or video display suitable for presenting 
images to the viewer 240 or for otherwise interacting with a 
human user, including liquid crystal displays (LCDs), pro- 
jection systems and displays based on cathode-ray tube 
technology. 

[0016] Generally, the viewer-location image compensa- 
tion system 200 optimizes the image for the current location 
of a single viewer 240 or an average location of all viewers 
240 in accordance with the present invention. The present 
invention optimizes an image for an oflset viewing location 
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of a viewer 240, where one or more of the pan angle, 0, tilt 
angle, <D, or distance, d, are outside a specified range of 
values. In this manner, the present invention employs image 
processing techniques to adjust the input image so that the 
output image appears as originally intended, for the current 
viewing location of the viewer 240. 

[0017] According to one feature of the present invention, 
the viewer-location image compensation system 200 adjusts 
an image to compensate for an oflset viewing angle of a 
viewer. In particular, as discussed further below in conjunc- 
tion with FIG. 3, the viewer-location image compensation 
system 200 morphs an image to compensate for an offset 
viewing pan angle, 0, or tilt angle, to compress portions 
' of the image nearest the viewer 240 and enlarge portions of 
the image further from the viewer 240. In this manner, the 
viewer-location image compensation system 200 allows an 
image viewed from an oflset viewing angle (0*0 or <P*0) to 
appear as if the image is viewed from a direct viewing angle 
(0 and <I> approximately equal to 0). 

[0018] According to another feature of the present inven- 
tion, the viewer-location image compensation system 200 
adjusts an image to compensate for a viewing distance, d, 
outside of an optimized viewing range of a display 230. In 
particular, as discussed further below in conjunction with 
FIG. 3, the viewer- location image compensation system 200 
changes the size of an image to compensate for a viewing 
location, d, outside of an optimized viewing range of a 
display 230 (cUd^ or d>d max ). 

[0019] Thus, if the current viewing distance, d, is greater 
than the optimized region (d>d max ), then the image is 
enlarged. Likewise, if the current viewing distance, d, is less 
than the optimized region (d<d min ), then the image is 
reduced. For example, the viewer-location image compen- 
sation system 200 can scale the image size to compensate for 
a viewing distance outside of the optimized viewing region. 
In an image having textual portions, for example, the size or 
thickness (or both) of the text can be adjusted. In this 
manner, the viewer-location image compensation system 
200 allows an image viewed from a viewing distance, d, 
outside of an optimized viewing range of a display 230 to 
appear as if the image is viewed from a viewing distance, d, 
within the optimized viewing range of a display 230. 
[0020] Each camera 250 may be embodied, for example, 
as a fixed or pan -tilt- zoom (PTZ) camera for capturing 
image or video information. The image information gener- 
ated by the camera(s) 250 are processed by the viewer- 
location image compensation system 200, in a manner 
discussed below in conjunction with FIG. 3, to determine 
the viewing location of a viewer 240. It is noted that a 
one-camera system can estimate the viewing distance, d, 
based on the size of the person appearing in the image 
(assuming a standard size person). 

[0021] The viewer-location image compensation system 
200 may be embodied as any computing device, such as a 
personal computer or workstation, that contains a processor 
220, such as a central processing unit (CPU), and memory 
210, such as RAM and/or ROM. Alternatively, the viewer- 
location image compensation system 200 may be embodied 
as an application specific integrated circuit (ASIC) (not 
shown) that is included, for example, in a television, set-top 
terminal or another electronic device. 
[0022] Memory 210 configures the processor 220 to 
implement the methods, steps, and functions disclosed 



herein. As shown in FIG. 2, the viewer-location image 
compensation system 200 includes an image adjustment 
process 300 that is implemented by the processor 220. 
Generally, the exemplary image adjustment process 300 
monitors the location of one or more viewers) 240 and 
dynamically adjusts the image to compensate for the current 
location of the viewer(s) 240 in accordance with the present 
invention. The image adjustment process 300 can optimize 
an image for the current viewing location (pan angle, 0, tilt 
angle, <I>, and distance, d) of a viewer 240. 

[0023] The memory 210 could be distributed or local and 
the processor 220 could be distributed or singular. The 
memory 210 could be implemented as an electrical, mag- 
netic or optica] memory, or any "combination of these or 
other types of storage devices. Moreover, the term 
"memory^* should be construed broadly enough to encom- 
pass any information able to be read from or written to an 
address in the addressable space accessed by processor 220. 
With this definition, information on a network is still within 
memory 210 because the processor 220 can retrieve the 
information from the network. It should be noted that each 
distributed processor that makes up processor 220 generally 
contains its own addressable memory space. 

[0024] FIG. 3 is a flow chart describing an exemplary 
image adjustment process 300. As previously indicated, the 
image adjustment process 300 monitors the location of one 
or more viewers) 240 and dynamically adjusts the image to 
compensate for the current location of the viewer(s) 240 in 
accordance with. the present invention. The image adjust- 
ment process 300 may be executed continuously, intermit- 
tently or upon a detected movement of a viewer 240, as 
would be apparent to a person of ordinary skill in the art. 

[0025] As shown in FIG. 3, the image adjustment process 
300 initially obtains one or more images from the camera(s) 
250 during step 310. Thereafter, the image adjustment 
process 300 determines the location of any viewer(s) 240 
that are present during step 320. A test is performed during 
step 330 to determine if the current viewing location of the 
viewer(s) 240 is within a predefined tolerance of specified 
values for each of the pan angle, 0, tilt angle, O, and 
distance, d. 

[0026] If it is determined during step 330 that the current 
viewing location of the viewer(s) 240 is not within a 
predefined tolerance of a specified viewing location, then the 
image is adjusted during step 340 to compensate for the 
oflset viewing angle or distance. An exemplary technique for 
adjusting the image to compensate for the offset viewing 
location of the viewer is described below in a section entitled 
"Image Adjustment Technique." 

[0027] If, however, it- is determined during step 330 that 
the current viewing location of the viewers) 240 is within 
a predefined tolerance of a specified viewing location, then 
program control terminates. 

Image Adjustment Technique 

[0028] The original image can be expressed as a two-by- 
two matrix of pixels. In order to compensate for an offset 
viewing location, the original image is adjusted in an exem- 
plary embodiment of the present invention using a linear 
transformation technique. Generally, the linear transforma- 
tion maps the pixels in the original image, I, to a new space 
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to generate a modified image, M, that distorts the image, 
such that when the modified image is viewed from an offset 
viewing location the image appears as if being viewed from 
a direct viewing location. Thus, a given pixel in the original 
image can be expressed as Pj and a given pixel in the 
modified image can be expressed as P M . 

[0029] As previously indicated, the current viewing loca- 
tion is the current location of the viewer's eye, P e , and is 
fully defined by the pan angle, 0, tilt angle, <I>, and distance, 
d, relative to a fixed point on the display, the current location 
of the viewer's eye, P e , can also be expressed as follows: 



where R = 



cos0 0 -sin0 
sirrf>cos0 cos* sirrf>cos0 
cos<t>sin0 -sixrf> cos<t>cos0 



[0030] In a first embodiment, it is assumed that the user is 
far away from the display. The distance from the display can 
thus be ignored. Thus, each pixel in the modified image, P M , 
can be obtained by identifying the appropriate index of a 
corresponding pixel in the original image, P r Thus, to obtain 
a pixel value in the modified image, the appropriate index of 
the corresponding pixel in the original image, P t , is identi- 
fied as follows: 



p,=R p„ = R 



[0031] Since this embodiment ignores the distance from 
the display, the corresponding pixel in the original image, P„ 
can be expressed as follows 



»■(;} 



[0032] In a second embodiment, the distance, d, of the user 
from the display is considered. Thus, to obtain a pixel value 
in the modified image, the appropriate index of the corre- 
sponding pixel in the original image, P x , is identified as 
follows: 



( <p e -p M ,p e > 



where P u = 



[0033] It is noted that in both the first and second embodi- 
ments, if the calculated index of the corresponding pixel in 



the original image, P It is not an integer value image inter- 
polation is used to obtain the pixel value at the appropriate 
pixel location. 

[0034] As is known in the art, the methods and apparatus 
discussed herein may be distributed as an article of manu- 
facture that itself comprises a computer-readable medium 
having computer-readable code means embodied thereon. 
The computer readable program code means is operable, in 
conjunction with a computer system to carry out all or some 
of the steps to perform the methods or create the apparatuses 
discussed herein. The computer-readable medium may be a 
recordable medium (e.g., floppy disks, hard drives, compact 
disks, or memory cards) or may be a transmission medium 
(e.g., a network comprising fiber-optics, the world-wide 
web, cables, or a wireless channel using time-division 
multiple access, code-division multiple access, or other 
radio-frequency channel). Any medium known or developed 
that can store information suitable for use with a computer 
system may be used. The computer-readable code means is 
any mechanism for allowing a computer to read instructions 
and data, such as magnetic variations on a magnetic medium 
or height variations on the surface of a compact disk. 

[0035] It is to be understood that the embodiments and 
variations shown and described herein are merely illustrative 
of the principles of this invention and that various modifi- 
cations may be implemented by those skilled in the art 
without departing from the scope and spirit of the invention. 

What is claimed is: 

1. A method for adjusting an image, comprising: 

determining a viewing location of a viewer of said image; 
and 

adjusting said image to compensate for a viewing location 
outside a predefined viewing range. 

2. The method of claim 1, wherein said viewing location 
is outside a predefined viewing angle range. 

3. The method of claim 1, wherein said viewing location 
is outside a predefined viewing distance range. 

4. The method of claim 1, wherein said adjusting step 
further comprises the step of mapping pixels in said image 
to a new image space using a linear transformation that 
creates a distorted image such that when said distorted 
image is viewed from an offset viewing location said image 
appears as if being viewed from a direct viewing location. 

5. The method of claim 4, wherein said linear transfor- 
mation morphs said image to compensate for an offset 
viewing angle. 

6. The method of claim 4, wherein said linear transfor- 
mation scales said image to compensate for a viewing 
distance outside said predefined viewing distance range. 

7. The method of claim 4, wherein said mapping of pixels 
to a new image space creates an image with a greater number 
of pixels using an interpolation technique. 

8. A method for adjusting an image, comprising: 

determining a viewing location of a viewer of said image; 
and 

mapping pixels in said image to a new image space using 
a linear transformation that creates a distorted image 
such that when said distorted image is viewed from an 
offset viewing location said image appears as if being 
viewed from a direct viewing location. 
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9. The method of claim 8, wherein said viewing location 
is outside a predefined viewing angle range. 

10. The method of claim 8, wherein said viewing location 
is outside a predefined viewing distance range. 

11. The method of claim 8, wherein said mapping morphs 
said image to compensate for an offset viewing angle. 

12. The method of claim 8, wherein said mapping scales 
said image to compensate for a viewing distance outside said 
predefined viewing distance range. 

13. The method of claim 8, wherein said mapping of 
pixels to a new image space creates an image with a greater 
number of pixels using an interpolation technique. 

14. A system for adjusting an image, comprising: 

a memory for storing computer readable code; and 

a processor operatively coupled to said memory (160), 
said processor configured to: 

determine a viewing location of a viewer of said image; 
and 

adjust said image to compensate for a viewing location 
outside a predefined viewing range. 

15. The system of claim 14, wherein said processor is 
further configured to map pixels in said image to a new 
image space using a linear transformation that creates a 
distorted image such that when said distorted image is 
viewed from an offset viewing location said image appears 
as if being viewed from a direct viewing location. 

16. The system of claim 15, wherein said new image 
space has a greater number of pixels obtained using an 
interpolation technique. 

17. A system for adjusting an image, comprising: 

a memory for storing computer readable code; and 

a processor operatively coupled to said memory (160), 
said processor configured to: 



determine a viewing location of a viewer of said image; 
and 

map pixels in said image to a new image space using a 
linear transformation that creates a distorted image 
such that when said distorted image is viewed from an 
ofiset viewing location said image appears as if being 
viewed from a direct viewing location. 

18. The system of claim 17, wherein said new image 
space has a greater number of pixels obtained using an 
interpolation technique. 

19. An article of manufacture for adjusting an image, 
comprising: 

a computer readable medium having computer readable 
code means embodied thereon, said computer readable 
program code means comprising: 

a step to determine a viewing location of a viewer of 
said image; and 

a step to adjust said image to compensate for a viewing 
location outside a predefined viewing range. 

20. An article of manufacture for adjusting an image, 
comprising: 

a computer readable medium having computer readable 
code means embodied thereon, said computer readable 
program code means comprising: 

a step to determine a viewing location of a viewer of 
said image; and 

a step to map pixels in said image to a new image space 
using a linear transformation that creates a distorted 
image such that when said distorted image is viewed 
from an offset viewing location said image appears 
as if being viewed from a direct viewing location. 
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